In many places they did. Particularly useful structures like fortifications where Roman forts and city walls were kept in as good a shape as possible and formed the basis of the defence of many cities like Exeter for a good while into the post-roman period of English history, hell even older structures like that such as hill forts were often refortified as the old earth works and the commanding positions they were constructed at still made them a handy place to build a new layer of defences on them. And at times they were also demolished to provide new building materials during this time. But the population of Saxon Britain in many places was smaller than it had been at the peak of Roman rule. The Romans invested quite a bit in particularly the miltary defence of the province which both had large numbers of legions stationed there (who would often be spending some of their pay locally, the army purchasing goods from local sources ect driving a monetary economy) and produced large numbers of auxiliary infantry (which might return home wealthier than they had left) and combined with things like the textiles industry that helped to strengthen the Province (or provinces later on when they began splitting large frontier provinces like Britain into smaller ones). Between successive plagues in the third century and sixth centuries and the wide spread destruction caused by conflict in the province where the fall of the Roman Empire was experienced particularly violently the population in those parts of the country most Romanised such in the south east with Essex, Sussex, Kent, London and in the North East surrounding places like York large scale estate style farming, large towns and cities popped up at various points during Roman rule, shifting more towards thriving market towns and estates as opposed to concentrated in a few cities after the crisis of the third century then a lot of this withered away after the Roman final withdraw from Britain, the Saxon invasion and the Plague of Justinian.
As a result the amount of labour for rebuilding, the amount of housing needed and amount of large scale public facilities being used was a fair bit less. Additionally the late roman state was extremely compared to the small relatively decentralised kingdoms throughout the British Ilse that existed after they left meaning there was less money to spend on things beyond necessitates.
Additionally, the Roman market, monetary economy and large scale well connected trade networks helped to support things like stone masons and skilled architects. With a much less monetary based economy in post Roman britain supporting such types of social complexity and paying people who have such professions was harder so while they there were certainly still people who knew how to build stuff there tended to be less of them with a smaller prompting of the already smaller population devoted to things beyond subsistence farming. As a result unless there was a pressing reason to people were less likely to go to the expense of hiring people who could work with the materials the Romans used and rebuilding on the scale they did.
Not sure if you've seen it, but if not you will definitely find this other r/askhistorians post interesting. I know I spent an embarassing amount of time reading it and I haven't even played the game.
There are a few threads (like this one) (edit: with answers by /u/thefeckamIdoing, /u/BRIStoneman, & /u/sagathain) that specifically address your question.