Disclaimer: The term "ancient world" obviously covers a number of civilisations, all of which need to be considered seperately.

I assume, your question hints at the statement that the Greeks invented mathematics, because they had slaves and hence enough spare time to think about such abstract things as mathematics. While greek mathematics indeed does not seem (not seem!!!) to bother with any applications, it is quite generally agreed, that the invention of numbers (and number systems), as well as writing in general, seem to have been developed due to administrative needs. This need is in fact the motor behind great parts of ancienct mathematics. Thus, mathematics seems to stem from applications and not necessarily a "thinking elite", but a "working elite".

We're obviously not talking about physical work, though. If you look at the articles by Annette Imhausen and Eleanor Robson about mathematics in Egypt and Babylonian civilisations in this book you will realize, that they usually discuss mathematical texts written for and by scribes, or scholars who were being educated to become scribes. This clearly points to the assumption that mathematics was a discipline practised by a small number of people, high standing officals, who were valued for their special skills. On the other hand, you need to bear in mind, that one of the main problems with the research of Egyptian mathematics is the lack of sources. Next to 3 major papyri (among them Papyrus Rhind), we only have some fragments or texts that provide indirect knowledge about mathematics in Egypt. Hence, it could theoretically be possible that everybody learned mathematics, maybe using a perishable material like sand. Still, we have no sources that bear the slightest indication that this was the case. (While we have significantly more sources on Babylonian mathematics, I'm not aware of a single one implying that mathematics was taught to the majority of the population).

Quite contrary, if you look at the existing sources, the scribes who are able to do mathematics clearly convey an image of self-proclaimed importance due to their skills that set them apart from the common man. Cf. this exerpt from the papyrus Anatasi:

"Hey Mapu, vigilant scribe, who is at the head of the soldiers, distinguished when you stand at the great gates, bowing beautifully under the balcony. A dispatch has come from the crown prince [...] An obelisk has been newly made [...] of 110 cubits in the length of its shaft, its pedestal of 10 cubits [...more parts]. Add them up in order to make it from parts."

So, while mathematics itself is not necessarily a proof of a thinking elite, the people who practised it, definitely are.