As usual, most of my knowledge is about the late empire rather than the early empire.
The short answer is, sorta, not really.
I say sorta because amongst the rich, they certainly did travel, and there was a limited phenomena of wanting to journey to important cult sites, whether "pagan" or Christian.
I say not really because you have to actually think very closely about what tourism actually is. The idea of western tourism originates much more closely from the medieval pilgrimage than late Roman travel. It is about the journey to a liminal space in order to have a close physical experience with the divine. However, that also presupposes that the places you're going to visit is divine. Think about the phenomena of the souvenir. It is in fact, a secular appropriation of the christian relic. A reminder of your journey to a liminal space where you had a life-changing encounter with something special.
The problem of this framework is it doesn't quite apply to the late Roman era of the 4th to the 5th centuries, and really only kicks off later in the 6th to the 7th century. A lot of current research wants to point out that records of journeys of what WE think are pilgrimage, did not actually have the same mental goal as later pilgrimages (Egeria's travels for example, or the Life of Melania). What that means is ascetics were traveling not to experience something liminal from a specific place, but were interested in traveling in order to "otherize" and isolate themselves (pilgrim is from the latin peregrinus which means wanderer) in order to recreate a monastic desert within themselves.
This is also helped by the fact that most Roman cities were completed on a template. The best way to think about what this means, is to think about the average non-descript US city: they all have the same franchises. If most of the cities near you follow the same template and have the same franchise, are you really being a tourist when you travel, or are you simply going to the same place but in a different location?
There are notable exceptions to this, namely that Rome was known as a city of splendor, or that Egypt was a land of antiquity, or that the Holy Land was the place where Jesus lived, but I really want to stress that at least within a Christian context, in the 4th and 5th centuries, the early centuries after the "legalization" of Christianization, the mental and monetary framework was not there to encourage "tourism" as we know it now.
And at the same time, there weren't networks of hostels/hospitals to support travel SPECIFICALLY to visit places. An emperor or a rich person might go to a city and see the local points of interest, but outside of religious reasons, they wouldn't be going specifically for that, and to reiterate what i just said before, even for religious reasons, it wasn't quite the same as later pilgrims.
Some reading: