The reason essentially boils down to a question of logistics and organisation. In the immediate aftermath of Imperial withdrawal, things continue largely as before, but as the mechanisms of state start to crumble as the province is beset by war and raiding and instability, so it becomes less and less possible to raise the vast amount of men and material necessary to maintain stone buildings. The Romans had been able to mobilise legions for military construction, and vast arrays f slaves and highly organised supply chains, all of which becomes increasingly difficult as the country splits into small, fragmented petty kingdoms.

The end of the Empire also brings around economic and limited population collapse. Procopius and Gildas both imply that the incoming Anglo-Saxons ravage and eradicate the Romano-British population, driving the survivors and refugees into Wales and overseas to Brittany, but DNA evidence suggests that instead, people simply abandoned the cities as the collapsing Roman economy and smaller kingdoms meant that returning to an agrarian economy is simply more viable. As fewer and fewer people live in the cities, there's simply nobody there to maintain them, and in some places, the building are looted for masonry to build rural farmhouses.

Following the establishment of the Anglo-Saxons in England, much of the actual technology to construct in stone is lost, until the Augustinian mission of 596. Alongside the return of the Catholic faith, the Church brings literacy, learning and science, and the technology to build in stone. One of the priorities of the mission is the restoration of the Church in Canterbury in stone, followed by the city's walls. This is a trend which then spreads across England, and as kingdoms grow larger and more powerful, and more able to muster the manpower and bureaucratic organisation necessary, so they build more and more in stone.

Charles-Edwards, T. ,After Rome

Bartlett, R. The Making of Europe