Why was the loss of three legions at the Battle of the Teutoburg Forest so devastating to the Roman military when the Romans had lost far more troops during previous wars, when the empire was smaller? Why did it bring Roman expansion into northern Europe to a halt?

You might find some elements of answer in these previous answer there ( u/LSCatilina) and there ( u/Celebreth).

The short answer would be that the conquest of Germania would have required more investment and effort than it would have be worth, politically in particular.

The campaigns in Germania before Teutoburg were already difficult enough and in spite of claims of having pacified the region between Rhine and Elbe, Romans and their auxiliaries were still threatened by a general hostility all the more dangerous that Rome couldn't be supported by regional structures (allies, road network, granaries, etc.) as they were in Gaul. Even if the provincialization was a priority for Augustus' visions of his rule, it remained largely dependent from Gallic supplies and Roman political support to justify the involvement in a quite remote and fairly poor region.

Critically, this general hostility wasn't challenged by Roman military victories against Germans that were aware of the weaknesses of their opponents (especially as some of them were former auxiliaries) : the Battle of Arbello years before Teutoburg had strong similarities with the later battle on this regard.

The loss of three entiere legions was an immense blow for Augustus and the credibility of campaigns whose difficulties had been be put asides and, so to speak, already celebrated as a "Mission Accomplished" by the reigns' cantors. Not only the region had to be re-"pacified", while it never really was, but the specter of renewed Germans raids across the Rhine and the Alps made a come-back, with a supposed threat in Gaul (where revolts of Belgians peoples were still of very fresh memory), in Illyricum and even Italy shocked a public opinion that was unable, contrary to what happened in Punic Wars, to form a "levée en masse" of citizen-soldiers with the current model of professionalisation of the imperial army.

It's not that Romans renounced to get the upper hand in Northern Europe : but besides the punitive expeditions under Augustus, new approaches were preferred, as these campaigns weren't profitless. They allowed a direct interaction between Romans and Germans, with a creation of alliances, exchanges, set of obligations that outlasted the abandonment of transRhenan lands : Romans goods found their way in trough state-backed trade and gift-diplomacy, further trying local chiefs. Thus, and backed by military "police" intervention or missioned advisors (as far as modern Silesia), Romans had access to German goods, recruitment of auxiliaries or mercenaries, controlled and seasonal migration, a certain influence over the border trough clientelisation of local chiefs, etc. All of this made the right-bank of the Rhine not that different from the Roman-controlled left-bank, with the agricultural and societal development of Rheinish societies in lines still dominated by the empire.

Eventually, the Rhine was far too convenient as a border, both strategically and logistically; while what Romans would have to pay up for conquering and provincializing Germania was certainly going to be extremely costly for dubious gains that they could get from political and military influence over Barbaricum.