More can always be said, but there's some recent scholarship putting it all in a bit of doubt. This paper mentions in passing that there is archaeological evidence that small pieces of cloth were used for anal hygiene. This short piece in German mentions that recent scholarship thinks the sponge on a stick was really used for anal cleaning, or whether it was more akin to a toilet brush. Finally, this paper by Wiplinger which is only available in German puts forth a theory that it really was more hygienic than the common theory of a sponge on a stick suggests. Since the paper is in German, I'll paraphrase the most important points:

• There is some archaeological evidence for small jugs that would be used much like a mobile low-tech bidet - one puts the jug below yourself and uses the left hand to clean, a method that is still in use in some Middle Eastern countries

• The evidence for the use of the sponge on a stick for anal hygiene was always rather weak - apparently there's only five sources and they are much less than clear. None of them point to the alleged usage, only three say the sponge on a stick was used in latrines

• The usage of a xylospongium is not only unhygienic, it also appears to be rather unpractical and even dangerous

• Now, the author proposes two other usages: Maybe it was used like a toilet brush, and the author suggests the common usage of such devices in many cultures nowadays indicates this may be the case. or maybe it was used to soak up excess water on the seats.

• The author also gives some evidence that many latrines were likely clean and well-maintained rooms, such as the availability of water to clean, and the usage of marble in upper-class latrines.

in sum, it would appear to me that there's little evidence for the usage of a shared sponge of a stick, and som evidence for alternative means of anal hygiene.

Also, as said, many of the latrines appear to have been of high standard, as the author linked above points out, whereas others may have not been as clean or beautiful - but this isn't different from today, the free public toilet in parks, for example, tend to be much less often cleaned than private toilets even today.

I'd also like to point out - heck, so does the author of the last link - that wikipedia somehow has an article saying the sponge on a stick method was definitely, certianly used, but the source is... questionable. Sure, there's a Seneca letter where he writes about a gladiator committing suicide by ramming a sponge on a stick down his throat in a latrine, but crucially, Seneca writes this device was used to remove the obscene. What this word means is questionable. Some translate it as the butt, or anal region, whereas others translate it as stain, a smear. Wiplinger points out that the word obscene isn't usually taken to mean the butt, but a bunch of unclean things. This does make it questionable that it was indeed used to clean, well, the butt.

Finally, I would like to point out that all of this is informed speculation, not certain knowledge. We don't have direct sources of a Roman author describing in-depth how latrines worked, and we don't have direct sources of how latrines were cleaned. However, I think Wiplinger's theory makes a lot more sense than the commonly repeated one about a sponge on a stick.