While that sort of scenario is definitely the case when the Romans come to Britain, their departure is quite a different beast. Even the withdrawal of the garrison in 410 is not as clear-cut as popular history maintains, and doesn't necessarily imply the end of Roman Britain. I wrote another answer here which goes into a little more detail about specifically the collapse of the Romano-British physical infrastructure which ties in quite well with the general theme.

Although the garrison withdraws from Britannia in 410, the provincial government doesn't go with it, and is largely left to be autonomous. Despite this, it tries to maintain links with the rest of the Empire for as long as possible. In the early-mid fifth century, the main threat to Britannia isn't from the Anglo-Saxons but from the Picts, Gaels, Irish and Scots. As the Roman military force vanishes, so raiding from the North and across the Irish Sea becomes increasingly problematic. The issue is compounded by the Roman military having taken their architects and engineers with them when they left, leaving few if any people capable of maintaining the series of Roman forts and outposts that could have offered some protection to the province. The Romano-British do try and resist as best they can, but of course have not been responsible for their own military organisation for several centuries at this point. On at least two separate occasions, Roman legions do in fact return to Britain to defeat particularly large raids or invasions and manage to restore order, but of course the situation deteriorates again once they leave. Eventually, however, according to Bede, the situation does stabilise and the Romano-British are able to restore a relative level of stability and prosperity. As you might have guessed, this doesn't last, as increasing isolation from the continent combined with a series of poor harvests and a particularly virulent plague leads to widespread upheaval and precipitates the futher abandonment of the cities. This in turn emboldens the Irish, Scots and Picts to resume raiding.

It is at this juncture, some decades after the initial Roman withdrawal, that the Anglo-Saxons appear on the scene. Following typical Roman doctrine of using "Barbarian" Limitanæ or Fœderati, the first waves of Anglo-Saxons are invited to Britain as mercenaries, ready defenders against Gaelic raiding. It's when the Romano-British authorities become increasingly lax about paying them and the Anglo-Saxons realise that they are the sum total of the British defences that things start to go truly sideways.