What prevented people in ancient Greece from turning multiple tablets in during the event of ostracism, thus illegally forcing a member of their society to leave the state for 10 years

Quick corrections before I proceed. Ostracisms, as far as I recall, are only attested in the democracy of ancient Athens, not any other Greek state, and only in the fifth century democracy, not the fourth. Those voting did not use tablets but ostraka, pot sherds, pieces of broken pottery upon which they (or someone else) wrote the name of the person they wanted ostracized. The process was a constitutional means of ending stasis, civil strife, so forcing someone to leave Attika was not illegal.

To your question: In Athens, all citizens were assigned to one of ten artificially-created phylai, or tribes. A fence with ten entrances picketing off a large area was erected in the agora. Each entrance to the voting enclosure was assigned to a tribe and was guarded by a magistrate (either one of the nine archons or the head of the thesmothetes), who verified that the voter was a member of the proper tribe, that he held only one ostrakon, and that he hadn't voted before.