Since the IIIrd century, the Armorican region knew similar issues than southern Britain, namely Saxon, Frankish and quite possibly Scottish piracy. The Tractus Armoricannum and Nervium, an equivalent of the Saxon Shore in modern Brittany and Normandy covered its defence; but as in Britain while there weren't devastating effects, it saw a certain economic retraction and an abandonment of villae, at least their reduction.
It's at this point that, beside Frankish laeti (defeated Franks settled in the region as a taskforce pool), people from Britain might have began to settle in Armorica. These Brittons, according to Gildas, were sent in Armorica by Magnus Maximus. Altough there is no contemporary, and few archaeological evidence to back it, this military Britton presence in the IVth century (and possibly even earlier) is quite plausible.
Enrolled in a military and regional role they already did in the island , they would have found there a land, not too dissimilar to theirs, with a people speaking either a Vulgar Latin closely related to their own and/or a Gaulish speech not dissimilar to their own Brythonic language.
The Osismii' territory (roughly corresponding to modern Finistère) is one of the main candidate for this first settlement, as it might have been subject to more severe attacks (including from Ireland) and it was arguably a land more firmly controlled by early medieval Brittons in the VIth while they were still in the process taking over the rest of modern Brittany . But there were probably other Brittons in Armorica even back then.

Such troops movements are attested by late Roman historians, such as Zosimus, who stressed that British armies were renowned because they managed to beat Barbarians, hence their usefulness in the mainland. This reputation might have played an important role in their acceptance in Gaul.

As the Rhenan limes broke down in 407, leading to the military abandonment of Britain in 410, the situation in Gaul deteriorated quickly : Frankish federates are overrun; Stilicho can't send troops as he have to defend Pannonia; Britain is militarily abandoned and Vandals, Suebi and Alans are virtually unnoposed for years of rampage; which arguably spared Armorica. Local revolts and usurper arise (some, as Jovian, with Barbarian support). Then Goths enter in southern Gaul.
While Ravenna recover its authority in Gaul by the late 410's, this authority is significantly damaged and some place seem to begin to act autonomously in facts, namely in this case, a good chunk of Armorica.
We simply don't know much of the region before the mid-460's, but in 461 the acts of the Council of Tours mention amongst included people Mansettus, "bishop of the Brittons". We don't know where his seat was located in Lugdunensis III but it's fairly plausible that these Brittons inhabited modern Finistère. They might have been augmented by migrants from Roman Britain, a province whom social-economic features utterly collapsed in the early Vth century with likely migrations from East (more dependent from Roman structures) to West, including in Armorica where kinsmen were already present.
As the situation degraded further with the end of Valentinian dynasty and the rise of warlordism in the Empire, Armorica was let to itself (once again, not unlike Britain) which might have increased the reliance over Britto-Roman migrants or hired muscle from impoverished Britain, especially in a context of renewed social revolt (the Bagaudae) in the 440's and the absence of a real imperial authority, after the death of Majorian, in Northern Gaul which allowed most federated Barbarian to expand at the expense of local Roman elites (but with the blessing of Ricimer, electing to ally with Barbarian warlords against their Roman counterparts)

Northern Gaul went trough important changes in the 460's : local warlord Aegidus died, leaving no clear leadership amongst the Romans and non-federated Barbarians as Alans.
If we consider that the Adovaricus mentioned in Angers by Gregory of Tours was indeed Odoacer sent by Ricimer to deal with unruly Alans (which served as watchmen over the strategic passes of Loire) and to assert Ravenne's power in Northern Gaul, it might by the catalyser of the replacement of Alans by Brittons, more trustworthy for what mattered Odoacer than local Romans. It appears,

Indeed, while militarily abandoned in the early Vth, post-imperial Britain still had ties with the mainland and the attraction of payed, imperial service shouldn't be neglected. According Sidonius Appolinaris, by the late 460's, Britto-Romans forces led by Riothamus were already established in the Lower and Middle Loire, accompanied by forces coming from post-imperial Britain in significant numbers.

It also indicated by a letter sent by Sidonius to Riothamus that he was considered by the senator-bishop as a legit authority to pass judgment upon this matter.

^(The bearer of this is an obscure and humble person, so harmless, insignificant, and helpless that he seems to invite his own discomfiture; his grievance is that the Bretons are secretly enticing his slaves away. Whether his indictment is a true one, I cannot say; but if you can only confront the parties and decide the matter on its merits, I think the unfortunate man may be able to make good his charge, if indeed a stranger from the country unarmed, abject and impecunious to boot, has ever a chance of a fair or kindly hearing against adversaries with all the advantages he lacks, arms, astuteness, turbulences, and the aggressive spirit of men backed by numerous friends)

Let's briefly note that Brittons taking (freeing?) slaves away could be seen as reinforcement trough local, destitute population.

These first waves, which might have been important enough for playing such a role in northern Gaul alongside Romans and Franks, was a military-based migration, backed by the strategical needs of the Late Empire.
While utterly defeated by Euric alongside Romans and Franks, these Brittons remained in Armorica while playing no further role in Gaul from this point onward.