It strikes me as odd that the primary language spoken in Switzerland is Germanic rather than romantic. Switzerland borders both northern Italy and southeast France, regions that primary speak romance languages, and was controlled by the Romans for almost as long as those regions.
True, Switzerland was conquered by Germans after the fall of the western Roman empire, but so were Spain, France, and Italy, and yet the vast majority of modern-day populations in those countries still speak romance languages.
Linguistic shifts along the Rhine during Late Antiquity and Early Middle-Ages is a complex phenomenon : the whole Rhineland in particular (along most of the border regions of the Late Empire in Europe) underwent enough changes that the current Romance/Germanic border is 150 to 200 km west or south of the limes, and modern regions as Flanders or Palatinate are crushingly Germanic-speaking in the same way most of Switzerland is.
You're right to intuit that this phenomenon didn't happen all of a sudden, as there is still Romance speakers (French and Romansch) neighboring regions where Barbarian kingdoms were established without leading to a shift of languages, at best a limited influence on phonology and lexicon. Nevertheless, germanization of modern Switzerland is essentially traced back to Late Antiquity and the collapse of the western Roman state in the Vth century.
Unfortunately, there is a relative lack of sources concerning the linguistics of the period in particular (or, more broadly, the regional history, meaning that we have to rely on onomastics and analysis of possible substrates and adstrates, and leaving us with more or less convincing speculations.
Late Roman upper Rhineland seems to have been significantly devastated during the Third Century Crisis : the border had been overrun by Barbarian raids leading to several communities shrinking up to being slowly abandoned (Avanticum) and being militarized (Augusta Raurica, Constantia, Vindonissa) on a smaller scale. In the aftermath of the raids, especially committed by a coalition known as Alemanni and the abandonment of the Decumate Fields that set the limes once more one the Rhine, it became again a militarized border region just as the rest of the alpine arc under Roman control.
Now the provinces of Maxima Sequania and Raetia weren't utterly destroyed either : even if population shrank, cities of Jura and Swiss Plateau recovered from the crisis as it happened in Gaul, and eventually became the new regional centers (Geneva being walled thanks to stone quarried from Augusta Raurica, for example). Rather than a desertification, altough the former lowlands agglomerations suffered importantly from the times, being considered as half-ruined by contemporary observers. Population was certainly probably largely latinophone by the Vth century, although Gaulish seems to have well survived there too, contrary to the rest of Gaul, way until the VIth century (with influence on later Germanic speeches). But already by then, you had Germanic-speakers too, settled or employed as auxiliaries in the border region, as it was the case in all the Rhineland (especially as Alemanni raids continued during the IVth century) : it is largely unknown how much and where these people would have been settled as laeti or foederati, but it would be likely that they'd have been at fairly strategically important places, such as crossroads or passageways in conformity with their role of border guards.
The crisis at the borders of the western Roman state (in Pannonia in the early 400's, along the Rhine in late 400's) might have had important consequences.
Garrisons were withdrawn regularly from the limes to reinforce Roman armies against Barbarians or usurpers, as early as 401 by Stilico : even if other troops were sent back to guard Raetia, they were (as the rest of the Roman military) increasingly made up of Barbarian troops moving in Romania, such as Burgundians in the 440's in Sapaudia but probably as well Alemmanic forces south of Danube. Contrary to Franks or Goths, tough, Alamans did not entierely integrated the late and post-imperial romanitas, less on their own volition (Alamans seem to have adopted a fair asset of Roman features as they settled in the Decumate Fields) but by an active imperial policy to prevent them being too powerful. This culminated with a factual political abandonment of the region by the mid-Vth century, not too dissimilar to what happened in Britain or the rest of Rhineland.
The general insecurity seems to have increased the populations movements to west and southern highlands, and although the former border towns remained continuously inhabited, the loss of a strong political authority let locals to fend and organize for themselves in what seem to have been a fairly chaotic situation : bishoprics weren't really settled before the late VIth century, and no regional authority emerging from it, being too remote from the strong successor states of the western Empire, but without falling under the authority of the various Alemanic rulers either. One of the stabler centers might have been the bishopric of Coire, whose bishops as their colleagues did in the rest of western Romania, had a certain temporal power over the immediate region (maybe getting some of their authority of being a relatively ancient religious establishment), but it's likely you had a lot of other local powers with other bishops, local warlords, etc. tied to each other up to a point.
Linguistically, the situation must have been fairly complex, without a territorial continuum to really speak of but various "pockets" of Romance (toponimy does point at their presence in all the broad region), Celtic and Germanic speakers interacting with each other.
Raetia passed from Ostrogothic to Frankish overlordship in the VIth century, which was accompanied with some stabilization of the political situation : the, relative, Christianisation of Alemanni led to the creation of a "Germanic" bishopric at Konstanz with an authority over most of depolulated areas of Raetia. In the same time, this century might have seen an actual migration from Germanic populations north of the Rhine and Danube in relations to Frankish success repelling raids from the East.
Whereas the upper valley of the Rhine seems to have been fairly tardily germanized (with topology hinting at a germanization of local roots rather than passing trough the second consonant shift), and while the period remains extremely obscure due to the lack of sources, it appears that the political stabilization of Raetia favored the integration of Romance and Celtic enclaves into a germanophone ensemble after a period of complex multilingualism, in the same time the stabilization of local powers on the plateau does paralleled the maintain of Gallo-Romance and Retho-Romance speeches : germanization might have been favoured having German-speaking communities being set on accessible emplacements and with facilities to trade/exchange/move within modern Switerland but also the Germanic-speaking areas; whereas highlands and swamps limited this expansion.
This process took time : while a significant Germanic-speaking migration in Valais had been proposed for the VIth century, it might not have ended up with a clear linguistic dominance until centuries thereafter. Raetia being unmistakably a peripheral region for Francia for most of the Early Middle-Ages (being ruled, at least in the VIIth and VIIIth centuries, by a local aristocratic dynasty, that ruled the region effectively independently), there's no evidence nor clue for a Romance settlement as it had been proposed for Frankish Rhineland. Even by the Classical Middle-Ages, the linguistic border wasn't settled in Switzerland, with a significant push of German-speaking areas westwards and southwards up until the modern delimitation coming from the XVIth century.
Even if this topic is both complex and obscure, some explanations can be proposed for the germanization of Upper Rhineland : settlements of German-speakers benefiting from a fragmented political ensemble, geographical facilities in expansion and connection with Germany and germanized Rhineland, etc. with pockets of Romance being swallowed up as being set in relatively depopulated regions.
(thanks to u/EnclavedMicrostate for translating maps' keys)