Manors had very few people so there was no introduction of new DNA into the system so over the generations people would have gotten disorders which would decrease the functionality of the manor. So how did they survive?
Manors? As in...fortifications, towers and castles?
Those definitely didn't have very few people they had lots of people and were often similar in size to a town and in many cases actually were the town. Even the smallest fortified towers would have stables, farms, blacksmiths and everything else.
Lennox tower is close to me and is a comparatively tiny fortified tower and yet it had facilities to forge its own cannon balls - one of the dies?/casts? still exists, it has been built in to the wall surrounding the tower ...
If you're just talking about estates then they had farms under their ownership as well as churches and sometime schools and the people that lived there would travel to towns just like anyone else would so it isn't like they were forced to breed with their cousins if that's what you mean?