I say that mountains are better for the military aspect of nations because it requires a madman to cross tall, icy peaks. Even when people tried, it often went disastrously for the invader's army (Hannibal crossing the Alps for reference). In contrast, rivers are hard to cover extensively, and unlike mountains, they require said garrisons in order to be secure.
I understand Caesar and others conquered these territories for prestige, but what stopped future rulers of Rome from returning these lands, same as Hadrian liberated Trajan's conquests?
I am not sure I quite get the question, there is no mountain range that lay between Gaul and Germania, the Romans would have to either conquer all the way to what, the Urals? Or they'd have to give up all of Gaul?
Rivers were very defensible barriers especially given the way the Roman's defended and influenced territories. The Romans control the crossings so any large Germanic army would need to either.
The only way for a Germanic army to breach the Rhine in the early empire (i.e. when these borders were set) would be to amass far more resources than the Romans which of course they could come near to. One of the main reasons the Romans are successful is their ability to out man everyone else.
I think you also slightly misinterpret the Rhine border. It was the limit of the province but not the end of the Roman Empire. The territory beyond the Rhine had a number of Roman forts and guardposts designed to provide intel/warning on Germanic troops and to exert Roman influence beyond the Rhine. Indeed the other side of the Rhine was not the enemy they were allies of Rome. Rome drew auxiliary troops from them and their farms supplied the Roman military bases on both sides of the Rhine. If I take Hadrien's wall for example (I'm more versed in those specifics) the Celts beyond the wall were heavily dependant on supplying meat in particular to the garrison along the wall. They had a symbiotic relationship. Through this trade they become more Romanised with signs of Roman wine, pottery and jewelry in their societies.
Roman "defence" was proactive too. They didn't just look at their own lands and ignore Germania, they actively interfered. They looked to use wealth and status to "divide and conquer" trying to ensure no Germanic chief grew too powerful. When gifts and diplomacy didn't work punitive raids by the legions would occur, though it must be said large punitive raids tended to have political motives, either an emporer seeking glory or an army mutinous and in need of booty.
So I hope we're in agreement that river crossings made the Romans pretty unassailable by large armies. The other threat would be smaller bands, Viking like handfuls of 10-70 men looking to raid and disappear back over the border. The measures I have outlined above made it hard to get any large ship or large number of ships over to the Rhine. The north south nature of the Rhine makes it easy to defend, in order to do a Viking raid and sail into Roman heartlands they would have to navigate down one of the the tributaries, all easily defendable. It leaves raiders/migrants with one real option, swimming across and making their way through the Roman province on foot. If they were to raid they would have to march to a settlement, load up with their riches and then march back and swim across all without being discovered by the Romans. And the Romans would discover them before if not after. The Romans had regular outposts along the Rhine with a handful of men at each, sometimes as low as just two soldiers. About a mile or two further back were cavalry detachments. The tactic is clear, a warning post to alert the cavalry who would use speed and training to hunt and destroy any bands.
On top of all this it must also be pointed out that there are additional non security reasons for creating your large military bases around a river. You are effectively creating a military town and a water supply for both drinking and waste disposal is pretty fundamental for large numbers of troops. Rivers also form good transport and communication lines keeping the forts in touch and allowing quick transportation of troops across the border.