I assume you're talking about Imperial Rome. It's important to remember the distinctions between a castle and a fort. The Romans from the Empire build lots of forts, but no castles, because forts met their needs whereas castles were unthought of and not really necessary. This is because a castle is not just a defensive fortress, it is also the home of a lord or nobleman. There was undoubtedly a Roman nobility, but during the Empire this nobility had no attacks to fear; they lived almost all of them either close to a city or a fort, and most importantly, there was a very big guarantee of peace throughout the empire. A nobleman living in Hispania could spend his whole life there in a luxurious villa with winegardens to his front porch and never see a barbarian. The frontlines of conflict were, for most Romans in the Empire, thousands of kilometres away at the Rhine, Danube and near Syria. There were the occasional civil wars and invasions, yes, but these were the exception proving the rule before the Crisis of the Third Century. Before this crisis, most Roman cities did not even have city walls, because they were completely unnecessary.

Compare this to medieval Europe: the continent is divided into thousands of petty fiefdoms, all ruled by a local duke or prince or baron, all of them controlling small armies at the best of times compared to any standard Roman army of multiple legions. Professional soldiers are rare and expensive, wars for small provocations with minimal gains of land and wealth are common. The nobility of this age could not live in luxurious villas, bathing in subterraneously heated baths and growing old in peace. In medieval Europe, being a nobleman meant needing a castle or serving someone who had one, because if you don't and the neighbouring lord wants your stuff you're done for. A castle is perfect for a lord in medieval Europe, precisily because it requires only a few (expensive!) soldiers to defend and he can live there in safety. Going back to Romans, it's unfeasible to imagine an entire legion crammed into any castle you can think of. And even if it does fit, there's the food problem. 5000 men eat away supplies a lot quicker than 50. The Romans preferred field battles, where they almost always had the upper hand because of their superior infantry, or sieging enemies instead of being besieged, because they had cool sieging techniques and could impregnate the most impenetrable of fortresses and castles (just look up the siege of Masada).

So to answer your question: the Romans did not build castles because neither their way of warfare nor the way of life of their nobles required them.