There are two dimensions to this question, really, the linguistic and the historical- not that the two aren't connected. And there's a few different possible answers.

For one, there have been plenty of suggestions over the years that instead of a great deal of loan words, the Celtic languages contributed to Old English in much less obvious ways. There's a special issue of English Language and Linguistics (vol. 13 issue 2 (2009), eds. Filppula and Klemola) on "Re-evaluating the Celtic Hypothesis" which gives some ideas- from the usage of 'do' to the dual paradigms of 'to be' in OE its contributors work under the assumption that there is a Celtic substratum to Old English, i.e. Brittonic (or in one case Old Irish) exerted pressures on the English language in ways other than vocabulary. This is less obvious to the casual observer and more the purview of linguists.

More upfront historical answers have been given. One thing to keep in mind is that the Middles Ages were a time of many, many invasions and greatly swaying ethnic and cultural identity. Thanks to 19th century historians and linguists who believed in concepts of racial purity and inherent racial and ethnic traits, we have in our popular imagination clearly set lines- 'these were the Anglo-Saxons, these were the Britons,' etc. etc. Yet around this same time period you see the Irish take over Pictish lands, the Britons take Armorica, and so on, and these areas took on the ethnic markers of their invading peoples. We're also talking about vast swathes of time here- it's literally hundreds of years between major historical records in Roman Britain/early Anglo-Saxon England, time for a lot of unrecorded change to occur.

That all being said, the invasion/migration of Anglo-Saxons does appear to be a uniquely extreme event in its one-sided cultural dominance. Brian Ward-Perkins has a good run down of this issue in his article 'Why Did the Anglo-Saxons Not Become More British?' (The English Historical Review Vol. 115, No. 462 (Jun., 2000), pp. 513-533). The two primary reasons he gives for this are good ones, as far as I can tell. The first is that the Britons were never fully conquered by the Anglo-Saxons. Celtic languages continued in Cornwall, Brittany, and most importantly Wales at this time, and it can be speculated that there was an animosity between those Britons who did not cave to the migrant/invading forces (the Welsh) and the Anglo-Saxons. In this idea, the persistence of strife pressured the native Romano-Brits to fully assimilate into Anglo-Saxon society and reject their Britishness (now Welshness). It may be important to note that 'Welsh' derives from the Old English wealas, meaning 'foreigner'- at this point Anglo-Saxon cultural domination was so strong that the common name for the Britons became the Anglo-Saxon word for 'foreigner,' despite the Anglo-Saxons' own migrant status. Ward-Perkins additionally shows legal codes that endow English people with greater economic benefits than Welsh, implying that there were two separate groups coexisting but that there were clear economic pressures on ethnic identity toward anglo-saxonization.

The second reason Ward-Perkins gives is that the fall of the Roman Empire and its influence in Britain caused a relatively fast collapse of Roman society- it's architecture, coinage, military support, and even language. Towns were largely abandoned as the Britons were forced by internal and external military pressures into more militarized tribal societies. There were no centralized cultural authorities at this time to preserve Brittonic. Even if there were, there already existed a class-based divide of Latin/Brittonic, as the Romano-British elite would've spoken Latin, and thus there would've been very little benefit to status for the Anglo-Saxons to embrace Brittonic (Latin, however, did make its way into Old English through a significant amount of loan words even before the conversion efforts at the end of the sixth century).

It's overall a complicated question, of course, and our relative lack of historical evidence in this period doesn't help (hence our much hated term 'the Dark Ages'), but hopefully that gives some background.

In most other settings of peoples being conquered by invaders, there's far more retention of the original languages.

I think you'll find that with a bigger sample size, this becomes increasingly untrue.

In any case, this is a case of language shift. Thomason & Kaufman (1988) show that in language shift, it's usually the case that if anything transfers over to the language being acquired (in this case, Old English), it will be phonological and grammatical features, rather than vocabulary. This isn't really a history question; it's more suited to the Q & A Post in /r/linguistics.

Thomason, Sarah Grey, and Terrence Kaufman. 1991. Language contact, creolization, and genetic linguistics. Berkeley, Calif: University of California Press.

It is true that not a single feature of British Celtic has been convincingly identified in the morphology and syntax of Old English; the two aspects of language where a historical linguist would expect to find influence from a substrate language, as adults who acquire a second language (especially in the context of the arrival of a new political/social elite over an indigenous population) as a rule find it more difficult to master pronunciation and sentence structure than learning new vocabulary. Indeed, think of a monolingual English speaker learning French; they'll have little effort learning French words but may struggle with pronouncing unfamiliar sounds like a uvular trill and will often substitute the closest approximate from English. Over time these substitutions solidify into concrete linguistic changes.

That's why it's actually not very surprising that British Celtic imparted only about ten loanwords into English, but also why it's very surprising that there is an absolute lack of Celtic influence on Old English syntax and sound system. This is likely because the British Celtic languages were regarded as very low-prestige dialects spoken by marginal people with the lowest social status between the onset of Anglo-Saxon colonisation and the Norman Conquest. The Norman Conquest is significant because the introduction of a new, Norman, ruling elite upset the previous binary between British Celtic and Anglo-Saxon languages; there is indeed evidence of Celtic influence on the Middle English language which was made possible by English's reduction in prestige, being replaced by Norman French.

The Celtic influence on English is responsible for the latter's grammatical simplification which typifies the English language in comparison to other more synthetic Indo-European languages to this day. For instance, the loss of grammatical cases is likely the result of Celtic influence; Old English had four inflectional cases but by the Middle English period, lost them all except for a vestigal genitive 's - all British Celtic languages also lacked inflectional cases. English also developed an auxiliary verb 'do' for emphatic purposes (I do love Askhistorians), a verbal construction that is also made use of by the British Celtic languages to emphasize another verb. Finally, English also developed a progressive present tense combining the verb 'to be' with a verbal noun (I am writing) which also appears to have been a linguistic construction borrowed from British Celtic.

This influence didn't happen overnight. What's more likely is that these developments took place at an earlier stage in the more Celtic north and west of England, but only appear in the written record with the collapse of Old English as a distinct prestige language. This geographical aspect raises another possibility for the seeming lack of Celtic influence on English vocabulary; it may be possible that Old English speakers first came into contact with people who spoke a very Latinized "Lowland" dialect of Celtic that was distinct from what's been described as "Highland" British Celtic, the parent language of modern Welsh, Breton & Cornish, if Celtic languages were even present in the south-east of England at all. Indeed, it seems likely that before the arrival of Old English, Latin developed in that area of England in much the same way it also did in Gaul, becoming widely adopted with Celtic dialects being spoken only in very marginal spaces.

Finally, there is a possible phonological Celtic influence on Old English, that is obscured by sound changes within the British Celtic languages themselves. If one were to look solely at the surviving British Celtic languages and contemporaneous ones like Old/Middle Welsh, any phonological influence on Old English would be completely obscured due to the fact that the British Celtic languages underwent significant sound changes beginning around 500/600AD. This means that the hypothetical Lowland Celtic dialects the Anglo-Saxons encountered may not even have resembled later Welsh or Cornish, making the potential influence a Celtic British language on English imperceptible if we were only to look at other British languages. However, it may be possible to identify a Celtic influence on Old English's sound system through an analysis of Old Irish, of all languages. This is possible if we accept Peter Schrijver's thesis that Old Irish is actually Proto-Irish that only arrived in Ireland sometime around the Roman Conquest of Britain: this would explain the complete lack of dialectical variation in Old Irish and a series of dramatic sound changes between 400-600 which could be explained by the rapid adoption of Old Irish by an indigenous population who previously spoke another, possibly non-Indo-European, language.

Schrijver asserts that Old Irish descends from the hypothetical Lowland British Celtic language that Old English speakers would have been in contact with, and points out that the sound systems of Old Irish and Old English are remarkably similar, through comparison of the processes of u-umlaut in Old English and palatalization in Old Irish. As well, Old Irish has parallels to Old English breaking/a-umlaut in its system of velarization, a byproduct of palatalization where consonants that are not palatalized are pronounced by bringing the tongue into near contact with the palate. There other correspondences between the two languages such as stress on initial syllables, the loss of unstressed syllables in the middle of words & loss or shortening of final unstressed syllables, richer vowel systems in stressed syllables compared to unstressed ones and presence of voiceless fricatives.

So in sum, the British Celtic language imparted so few loanwords into English because 1) linguistic contact often changes syntax & morphology, not vocabulary 2) Celtic languages influenced English on two occasions: a phonological influence imparted by a hypothetical Lowland British Celtic language which can be discerned in Old Irish, its closest surviving relative. Celtic didn't influence written Old English syntax & morphology due to socio-linguistic reasons, but did influence Middle English grammar as a result of other socio-linguistic phenomena 3) the dialects spoken by Britons in the initial area of Anglo-Saxon colonisation may not have been Celtic at all (some form of British Latin) or a heavily Latinized form of Celtic distinct from other Insular Celtic languages. This is why Celtic influence on the English language is incredibly difficult to discern if we assume that the Highland British languages (Welsh, Breton, Cornish) are paradigmatic of British Celtic languages as a whole.

Wow people, excellent answers! Threads like this one are the reason why I do love AskHistorians ;)

Follow-up: Is the Belgae hypothesis (i.e., that Britain had a considerable pre-Anglo-Saxon Germanic speaking population, which would supposedly explain at least partially why the Celtic lexical influence on English has been so minimal) taken at all seriously by historians in this field? It seems dubious to me, but I'd like to know the opinion of persons more knowledgeable than myself.