Probably the same issues that prevented the Medieval English from doing the same. Scotland is actually quite a hard place to completely subdue, due to numerous reasons which include its difficult terrain. The Highlands are notoriously difficult to traverse, and much more so with an entire army complete with logistics train and everything else. What's more, it was a relatively low-population area without intensive grain agriculture - this is crucial as it deprived the Romans of towns/cities to take as places to rest and recoup, and there wasn't much grain to requisition in the upland areas.

All that said, the Romans in Britain were already at the end of the supply line as it were, and an expedition required fairly substantial resources to succeed. What I mean is this - if Rome had completely conquered highland Scotland, then rebellions or other threats would have been that much more difficult to deal with as they were at the far, far end of logistical networks. For instance, when the Batavians under Civilis rebelled against Roman authority, it was a massive problem. But in the end the Romans could call on resources from the entire western portion of the Empire (including Britain itself) to combat the threat. In Britain there were no forces surrounding it to lend help if needed. Thus the creation of fortified barriers to give the forces there a defensive advantage.

The historical argument I see the most today however is that there was indeed little reason for the Romans to conquer Caledonia anyway. It was a relatively unpopulated, hostile, infertile area that was almost prohibitively difficult to traverse, and was located at the very outskirts of an already backwater province. There was more to be gained by having friendly buffer states there who would trade, help prevent raids and generally do most of the work for them. This was the case in other areas as well - one could ask why the Romans didn't push their border to the Elbe, or conquer more of the Saharan desert. Because they would put more resources into doing so than they could reasonably expect to receive in taxes, raw resources and manpower.

I will add more information when I'm not on my phone, but my primary resource was I.A. Richmond's Roman Britain