No one knows, it's unclear in the Quran, the Hadith are ambiguous, and later sources provide a variety of answers. Hadith, for instance, say:

The word Saba'a means "The one who has deserted his old religion and embraced a new religion." Abul 'Ailya said, "The Sabis are a sect of people of the Scripture who recite the Book of Psalms."

It's unclear who uses Psalms but is not Christian or Jewish. In other places, it appears as a sect in Arabia that is remarkably similar to Islam (there are instances where others mistake Muslims for Sabian in the Hadith); therefore, we can safely assume they were not some faraway little known group to the Salaf. Instead, they're just little know to us. A later source reiterates that they use Psalms, and adds their founding prophet is Nuh/Noah. Again, this matches no known group.

This ambiguity has been used to mark certain groups as licit or tolerated within various Islamic states. I believe even Hindus in India were sometimes called Sabians by local rulers so as to not be obligated to treat them as pagans. It would make sense if the actual historical group referred the Mandaeans or a small surviving Jewish Christian sect or simply unaffiliated monotheists, but it's also possible that the term referred to someone else entirely. It's also possible that their association with Psalms is erroneous, and attributed simply because that way such that the three main tolerated groups (with Magi/Zoroastrian being a fourth, separately considered group) would be, associated with three of previously given holy books. That is, the Sabians have the Zabur/Psalms given to David, just as the Jews had the Tewrat/Torah given to Moses, just as the Christians had the Injeel/either Gospels or full New Testament given to Jesus, just as the Muslims have the Quran given to Mohammed (there are also the "Scrolls of Abraham" mentioned in the Quran but not associated with a group; it's very ambiguous what that could refer to).

Related question: I've heard that the people of Harran in Syria had themselves identified as Sabian in order to hold onto their polytheistic beliefs. What is the likelihood that they are the Sabians of the Qur'an and what other religions are eligible for that title?