For the most part they didn't. In most medieval villages, there were no structures made of stone or brick aside from the village church, although often that was not stone either. There also weren't stone walls, instead people encouraged hedges to grow at borders between fields. These are called hedgerows, and they worked quite well to mark boundaries, keep livestock in fields, and even prevent tanks from advancing on Germany in the twentieth century. Individual farmers didn't have time to collect the amount of rocks it would take to build a structure.

When there was stone construction, it came from the same places that all stone came from. Either quarries, or re-purposed from older Roman construction. This meant that to have a building, wall or bridge built from stone, whoever constructed it had to have money. This means that the church and towns collectively were more likely to have the money for stone construction. Eventually, you start seeing stone mills, because millers tended to have a bit more money than the average peasant.

My answer is not exactly about peasants, but in medieval and early modern Rome it wasn't uncommon to spoil ancient ruins of their marbles/stone so as to reuse them somewhere else. There was an entire road - today's via delle Fornaci - which was lined up with ovens where countless statues, inscriptions and whatnot were baked in order to get mortar (hence the name). Also, many houses and shops were built inside the ruins or relied on pre-existing structures for architectural support, think of the Fora or the theatre of Marcellus; but even the Pantheon's colonnade was encroached by small houses until the 15th century!

Noblemen, on the other hand.... did the exact same thing, and families such as the Frangipane or the Contis turned many ancient structures (such as Hadrian's mausoleum or part of the homonymous markets) into fortresses.