Before getting into the meat of my answer, I would like to note that just because there was no equivalent to the modern police force before the early 19th century, that does not mean there was no such thing as law enforcement. Towns across early America had constables that acted as law enforcement officers, and sheriffs had the power to deputize laymen in aid of enforcing the law (so called "posses," which you're familiar with if you've ever seen a Western). While law enforcement was not as thorough or systematized as it would be with the advent of modern policing, law enforcement definitely existed.
Until 1961, the 4th Amendment was only applicable to federal law enforcement officials. In 1961, the Supreme Court decided the case of Mapp v. Ohio, 367 U.S. 643 (1961), which applied the 4th Amendment's protections against the states. Before Mapp, then, the 4th Amendment's scope was seriously limited, as the vast majority of criminal law enforcement in America is done by state officials, not federal ones. So who were the federal law enforcement officials in the 1800s that were limited by the 4th Amendment?
It's important to understand that scope of federal criminal law is far larger today than it was in the 19th century (though even today, it's far smaller than state criminal law). Federal crimes were extremely limited in scope; the first federal law to criminalize anything was passed in 1790, and it gives a pretty good idea of how small the federal government considered its law enforcement duties. The crimes listed include piracy, treason, counterfeiting, murders occurring on federal lands, and various acts that impaired the integrity of the federal judiciary. This small scope is because the federal government in America is a government of enumerated powers: it can only pass laws consistent with powers granted to it in the Constitution. Today, the Commerce Clause of Article I, which grants Congress the power to pass laws regulating interstate commerce, has been interpreted very broadly by the Supreme Court, giving rise to a huge number of federal crimes. That broad interpretation was not present in the 19th century, and the role of interstate commerce was less prominent in a pre-industrial America. Therefore, the things that Congress did criminalize tended to be related to other, more specific powers, such as the power to coin money (thus, laws against counterfeiting) and the power to establish a postal system (hence, mail fraud). The scope of federally criminalized activity expanded somewhat after the Civil War to enforce Reconstruction laws, which I will discuss, but it was still quite small by today's standards.
Thus, federal law enforcement officials were those charged with enforcing federal criminal law. This includes most famously the U.S. Marshals, who were established in 1789 as the chief law enforcement agents of the federal government. Among their duties was enforcing laws against counterfeit money until the creation of the Secret Service in 1865. They also enforced the infamous Fugitive Slave Act of 1850, which controversially compelled private citizens to assist the Marshals in the capture and return of escaped enslaved persons. Federal law enforcement also included the United States Postal Inspection Service and its predecessors, which enforces laws related to postal crimes. The United States Customs Service enforced customs laws (i.e., laws relating to import tariffs), a hugely important office in the 19th century when the main source of federal revenue was tariffs rather than income taxes.
Though there were a number of different federal law enforcement agencies, the reality was that there just weren't that many federal law enforcement officers. In fact, in 1842, the Chief Justice of the United States, Roger Taney, lamented that federal law enforcement alone would be totally ineffective in helping slaveowners recapture escaped slaves, considering how few of them there were and how limited their geographic reach was. This is part of the reason why private citizens were compelled to assist Marshals enforce the Fugitive Slave Act--there just weren't enough Marshals to enforce the Act as well as slaveowners wanted, so private citizens were essentially forced to provide them with extra manpower.
Federal law enforcement would expand somewhat after the Civil War, as Reconstruction and a newly active federal government meant more scope for federal law. The Department of Justice was established in 1870, centralizing a previously scattered federal prosecutorial system. Congress passed the Enforcement Acts in 1870 and 1871, which criminalized race-based discrimination in voting by state officials, election fraud, bribery and intimidation of voters, and conspiracies to prevent citizens from exercising their constitutional rights. The Ku Klux Klan Act of 1871 famously criminalized conspiracies to deprive citizens of the right to vote, hold office, and enjoy equal protection of the laws--a rather broad and innovative statute passed under the auspices of the relatively new 14th Amendment, intended to go after, you guessed it, the Klan. The U.S. Marshals would enforce this law, and U.S. Attorneys employed by the Justice Department would prosecute them. The Marshals were extremely effective in breaking the Klan--by 1872, the Klan had virtually disappeared (though the end of Reconstruction and the reluctance of the federal government to continue aggressive prosecution of racially motivated terrorism meant that white supremacist paramilitary groups would reappear in the next decade or so, reestablishing white dominance and eventually disenfranchising blacks). Other changes in federal law, such as the criminalization of bigamy in the federal territories (in response to the early practices of the LDS Church in Utah) and a federal tax on alcohol (which affected moonshiners) also expanded the scope of federal law as the 19th century wore on. The Marshals would even engage in strike-breaking on behalf of private companies in the 1894 Pullman Strike, ostensibly on the grounds of the constitutional duty to maintain the mail system.
Broadly speaking, as the federal government oversaw a more industrialized, wealthy, and centralized country through the post-Civil War era, it expanded its criminal law enforcement powers. The scope of federal law enforcement at this time paled in comparison to the modern federal law enforcement apparatus, and was minuscule compared to state law enforcement. The vast, vast majority of individuals charged with crimes in the 19th century would never enjoy the protection of the 4th Amendment because they would be prosecuted by states, and the 4th Amendment simply was not applicable to state action until 1961.
I hope that's a sufficient answer!
Sources:
Allen et al., Comprehensive Criminal Procedure (4th ed. 2016)
Eric Foner, Reconstruction: America's Unfinished Revolution (1988)
Mapp v. Ohio, 363 U.S. 643 (1961)
Stephen Cresswell, Mormons & Cowboys, Moonshiners & Klansmen (1991)
Gautham Rao, The Federal Posse Comitatus Doctrine: Slavery, Compulsion, and Statecraft in Mid Nineteenth-Century America, 26 Law and History Review 1 (2008)
David Kopel, The Posse Comitatus And The Office Of Sheriff: Armed Citizens Summoned To The Aid Of Law Enforcement, 104 J. Crim. L. & Criminology 761 (2015)