(mainly in Rome, Greece, Egypt, etc.. but all info is appreciated)
So I realise that peasants probably couldnt afford it since they were poor and had to take care for their farms/families etc. But did the rich guys travel somewhere like we today do ?(to see some city/temple [not because of religious reasons]/ for entertainment ...)
Certainly. Pausanias's Description of Greece was probably an actual guide book for tourists. It dates to the Second Century AD and describes various sites grouped by region (Attica, Arcadia, etc.). He writes about various famous sites and artworks and transcribes inscriptions he (probably) read himself on his travels. One inscription he writes about, regarding the famous artist Praxiteles, has been suggested to have been put up during the time of Hadrian specifically for the benefit of Roman tourists.
These tourists - who would primarily have been of the upper classes - would have been interested in famous sculptures, buildings such as temples, and even large public works such as ports and harbours. Pausanias talks about the sites as well as what he regarded as their history or mythological backgrounds. At times he has an antiquarian feel about him, as he likes to talk about unusual things and not only the most popular things to see.
Sources: Pausanias; James George Frazer's Commentary on Pausanias's Description of Greece
Another form of tourism that the previous commenters have not mentioned is pilgrimage. Gods in the ancient world had their special places. Apollo had important centers in Delphi and Delos. Aesculapius's most important site was Epidauros. Much of what Pausanias (see /u/BaffledPlato's post) presents was useful information for pilgrims. Jerusalem was a huge draw for Christians, especially after Constantine created an entire district around the Holy Sepulchre (it would have been like a religious theme-park almost). We have multiple descriptions of ancient Christian pilgrimages, the oldest is the Bordeaux Itinerary.
I got interested in this topic a few years ago and read "Travel in the Ancient World" by Lionel Casson.
It's an excellent read on this subject and I highly recommend it. It covers motivations for travel (in particular travel not related to trade or government business, i.e. tourism or religiously motivated travel), facilities afforded to travelers (inns, bars, restaurants), and modes of travel.
I particularly enjoyed a section that quoted some ancient letters. Like this one to Theon in Alexandria from his son:
"A fine thing you did! You didn't take me with you to the city! If you don't want to take me with you to Alexandria, I won't write you a letter, I won't talk to you, I won't say Hello to you even."