But why would everyone surrender the comfort of living in cities and even Stone Houses etc.?

Most people weren't living in stone houses anyway, or if they were, not in the neat and tidy domus that we like to imagine. For the majority of the Roman urban population, home was an insula (lit. 'island), in essence a block of flats, typically cramped, over-crowded, usually in poor repair and prone to fire and collapse. Slumlord landlords were a popular target for Roman satire, with Cicero laying the accusation that many purposely skimped on construction costs and maintenance so that when the building inevitably collapsed, they could raise the rent for tenants to live in a new building. In a rural setting, the majority of people lived, essentially, in wooden buildings, with only the wealthy landowners typically owning the vast villa complexes of popular culture.

It's not like 5th Century Britons were abandoning the cities out of a new-found fashion for country living. While Gildas' narrative of English conquest isn't broadly supported by the evidence, the wider contemporary context is. The collapse of the Roman Imperial logistical network didn't just limit the supply of luxuries, it required a complete refocusing of the agrarian economy, and also impacted the pan-Imperial slave network, and in Britain, the removal of the Roman garrison forces left the isles increasingly vulnerable to raiding. While a sub-Roman state (or states) was able to emerge with some semblance of stability and prosperity, according to Gildas, the 420s were marked by an outbreak of plague so virulent that the living were unable to bury the dead. Cities, with their high populations in densely-packed tenements and unhygienic bath houses, were already prime centres of disease, and were similarly devastated by plague. City living is also only fine as long as there's a reliable food supply, but the plague purportedly brought with it famine as well as raiding. In short, people abandoned the cities in fairly short order once the food stopped coming and the corpses started piling up in the streets.

This didn't necessarily mean people stopped building in stone; In areas where stone was a readily available resource, people continued to live in stone-built houses, such as Dartmoor. Similarly, crumbling stone buildings in cities were frequently raided for ready building materials. Following the Augustinian Mission of the late 6th Century, the English proved themselves very accomplished stone builders, but limited this largely to religious buildings and defensive structures. While they were more than capable of building in stone, their cultural affinity remained for wooden buildings. Wood makes a lot of sense; it's a plentiful, cheap and renewable material that's much faster to work with than stone, and enables fairly rapid construction with good insulation.