Higham and Ryan have a fairly lengthy but accessible discussion of this topic in their The Anglo-Saxon World (2013). In essence, they argue that yes, it was because of the Anglo-Saxons. The reasons are not well understood, but the underpinnings of Romano-British society just did not persist as they did on the Continent after the arrival of the Anglo-Saxons. H&R are careful to make clear that we don't have a clear picture of what was going on, and that the events seem to be different depending on what region of Britain you examine. What they do say is that late Roman Britain had certain parameters: a society with separate roles for military and civilian functions; professionalized, institutionalized process of law with courts in the Roman style; a system of individual land tenure familiar to Roman Imperial practice; a robust system of patronage within a stratified social hierarchy; and an over-arching Christian religious framework. They assert that all of these things were overturned by Anglo-Saxon domination, and that this wholesale and comprehensive overturning of the status quo was not the case in Gaul (or presumably elsewhere on the Continent). They note that Latin lost its prestige status seemingly very quickly to Old English (and I'll note here that Latin's status as a prestige language and the necessary vehicle of "making it" in the Roman system is the primary reason why non-Romans were speaking Latin in Britain at all, in any time period). The system of Roman villa life as an organizing factor in the countryside of Britain disappears very rapidly in the early 5th century, and with that change comes fundamental changes to the rural organization of Britain, with sweeping socio-economic consequences. "Roman" towns are abandoned, for whatever reason, and new "Anglo-Saxon" towns appear, with their distinctive architecture so indicative of new social organization and so distinctively not-Roman. H&R go on: "The institutional framework of late Roman Britain dissolved into a society in which military capacity was embedded within the rank of freemen, linked together via the warrior band and kindred to maintain order. This reintegration of military capacity and the control of land was arguably the most significant aspect of the transition from Roman Britain to Anglo-Saxon England" (108).

They also talk about language as an indicator of ethnicity early on in Anglo-Saxon Britain:

In circumstances where freedom at law, acceptance within the kindred, access to patronage, and the use and possession of weapons were all exclusive to those who could claim Germanic descent, then speaking Old English without Latin or Brittonic inflection had considerable value. The leaders of Anglo-Saxon households therefore had a vested interest in resisting the adoption of Celtic or Latin loanwords, structural influence and/or phonetic change, and even the use of non-English names for their own settlements, in case such might imply ‘British-ness’ in the occupants. Self-interest therefore probably reduced the impact of earlier insular languages on Old English. By adopting unaccented English, lower-status individuals gradually established themselves as ‘Anglo-Saxon’, and opened up opportunities for themselves and their descendants. (110)

And they further note that in Britain, the Romano-Christian localization of citizenship and religious identity in the individual was not retained, and instead replaced wholesale with an emphasis on the family, "with personal status subordinated to that of the lineage and kindred." This "centrality of kin" contrasts with the early adoption of Romano-Christian ideas of individuality by Germanic kings on the Continent. The best parallels for how Roman Britain became Anglo-Saxon England in fact do not come from Spain, Gaul, or Italy, but instead the Balkans, where Slavic similarly quickly took over and with it a new society scrubbed of social hierarchy.