It was accepted for some time during the Roman Empire, before Christianity came in. I cannot speak nor provide any sources for Ancient Greece, although my Greek teacher says that there's a lot of hype around this subject and that pederasty was socially accepted in Greece only for short periods of time, and not everywhere.

Generalized, veiled, and under the covers social acceptance of prostitution dates back to the Late Middle Ages.

However, things were never like they are today. I would be able to give a better answer to your question if you had given a better definition of what you exactly mean by casual sex. Some tribes in Europe had the custom of wife sharing and wife exchanging, and during feudal times ( probably ) existed the jus primae noctis. I don't know if you'd call these things "casual sex".

Anyway, here are some sources.

Antiquity:

History of Private Life, Volume I: From Pagan Rome to Byzantium. Ariès, Phillippe Duby, Georges Veyne, Paul Goldhammer, Arthur.

Middle Ages:

Sex, Dissidence and Damnation: Minority Groups in the Middle Ages. By Jeffrey Richards.

History of Private Life, Volume II: Revelations of the Medieval World.