There is an amusing and very relevant quote by an Arab traveler visiting 9th century China describing his reaction towards toilet paper: "They (the Chinese) are not careful about cleanliness, and they do not wash themselves with water when they have done their necessities; but they only wipe themselves with paper." People used stones, rags, leaves, their hands, or anything they could find before the global spread of toilet paper. Some cultures considered the use of hands and water as more sanitary than paper.

Renaudat, Eusebius. Ancient accounts of India and China. Paris, 1718.

Archaeologists studying the sewers associated with a large housing block in Pompeii recently didn't find any traces of sponge, what they found instead was lots of torn up rags. It's only one housing block but it suggests the sponge-on-stick thing may be a myth and that Romans may have just wiped themselves with whatever was to hand.

I'm not linking to the interview with Mary Beard that I heard this in as it's 18+ material, but here's an article on a book in which she writes about it.

http://dovegreyreader.typepad.com/dovegreyreader_scribbles/2008/10/pompeii-the-lif.html

In India, they still use water for this purpose. It is not uncommon for a house hold in India not to have toilet paper at all.