I specialize in the colonial period, so I can tell you how American elections operated in a slightly earlier era (say 1700-1775). And elections in the Early American Republic would have been very similar, as states borrowed almost all of their election procedures from this earlier period. I have already given some information on colonial elections in this response. But let me address your questions about ineligible voters in particular.

Most colonies/states did have property requirements for voting: typically, about 50-100 acres. But these requirements were often not enforced very strictly. For example, in an election in Richmond County, Virginia in 1735, more than half of the voters did not meet the technical property requirements, but they were still allowed to cast a ballot. Generally, the way these rules were enforced was up to the discretion of an official called an election inspector or a returning officer. If the election inspector let you vote, you could vote. If not, too bad.

As you might imagine, the process of choosing election inspectors could get quite political in of itself, as these officials had a lot of power in determining the outcome of the election. In fact, there are examples of violence associated with choosing inspectors. In Philadelphia, election inspectors were traditionally chosen by having eligible voters line up behind their preferred candidate on the morning of the election. In an election in 1742, however, one party (the Proprietary Party) accused another party (the Quaker Party) of physically blocking the stairs to a particular candidate for inspector, thereby preventing his supporters from lining up behind him. In retaliation, the Proprietary Party's supporters began physically assaulting the Quaker Party's supporters, swinging clubs to clear them from the stairs. Ultimately, this process of choosing an inspector turned into a full-blown riot.

So yes, there was a process in place for determining voter eligibility, but this process was often somewhat unevenly enforced and could even occasionally be violent.

Sources:

Richard Beeman, The Varieties of Political Experience in Eighteenth-Century America (2006)

Michael Bradley McCoy, "Absconding Servants, Anxious Germans, and Angry Sailors: The Making of the Philadelphia Election Riot of 1742" Pennsylvania History, vol. 74, no. 4 (2007)