I can't give a full answer to the main question, but I can clarify some things around the edges. Several translations read "sandcastle", and rightly so to the extent that it makes the image immediate for a modern reader. But "sandcastle" is more specific than what the Greek says. Here's a more literal translation:

... ἔρειπε δὲ τεῖχος Ἀχαιῶν
ῥεῖα μάλ᾽, ὡς ὅτε τις ψάμαθον πάϊς ἄγχι θαλάσσης,
ὅς τ᾽ ἐπεὶ οὖν ποιήσῃ ἀθύρματα νηπιέῃσιν
ἂψ αὖτις συνέχευε ποσὶν καὶ χερσὶν ἀθύρων.

... he threw down the Achaians' wall
very easily, like when some child (does that to) sand by the sea,
one who made it toys in childish behaviour,
then straight after, scatters it randomly with hands and feet.

The words that I've translated as "toy" and "randomly" aren't everyday Homeric vocabulary. Richard Janko's commentary on the passage doesn't cover leisure activities, except to point out that this simile follows up on the theme of another simile three lines earlier, where Apollo makes a path as wide as a javelin cast by an athlete: both similes use leisure to counterpoint the military violence.

The Romans certainly liked sailing and swimming at the seaside, and had resorts and luxury seaside villas. Pliny the Younger's famous dolphin letter (Letters 9.33) mentions that people enjoy "fishing, sailing, and swimming" at the seaside near Hippo Regius (in northeast Algeria), and compete to see who can swim the furthest from shore. You'll find plenty of history books on Roman seaside leisure. But I'm finding it much harder to track down info about the Greeks. Sorry I can't be more help.

The history of the sandcastle only appears to go back to the Victorian reinvention of seaside leisure in the 1800s, in conjunction with the development of public transport. The word "sand castle", "sand-castle", "sandcastle" first appears in English in 1854 according to the OED.