As it happens, there is no direct mention of a right to privacy in the U.S. Constitution, amendments, or federal law although you are right that in many legal contexts privacy is talked about as a right. Probably the most interesting part of the discussion to be had here are all the different contexts in which something called privacy might come up and how it is conceptualized slightly differently in each but that's probably more interesting from a legal/philosophical view than a historical one. What's important to recognize is that recognizing a right is done by legislation or the courts and, in the US it was done by the courts.

The first really significant legal salvo in this direction was The Right to Privacy an article by Samuel D. Warren and the (incredibly influential and respected) Louis D. Brandeis published in the Harvard Law Review, Vol. 4, No. 5 (Dec. 15, 1890).

"It is our purpose to consider whether the existing law affords a principle which can properly be invoked to protect the privacy of the individual; and, if it does, what the nature and extent of such protection is." id. at 197. Before concluding that the law did, already, protect privacy the authors spoke of the "necessity" of privacy rights as the modern world of the 1890s was too hectic, with a press too invasive, for a person to navigate without some private retreat. As happens often in common law legal systems, the authors looked at prior case law for their point, largely drawing the right to privacy from property rights cases, specifically ones where someone asked the courts to prevent the dissemination of images taken of them (among these being Prince Albert [yes, THAT Prince Albert] v. Strange, 1 McN. & G. 25 (1849) regarding some private etchings the mere possession thereof was seen as a breach of privacy).

I'm not sure how interesting the various legal arguments they trace out are to you (can go into that if you'd like) but, to oversimplify almost to the point of taking it out of the legal realm, the article concluded at various points that while courts claimed to be resting their decisions on a property rights to images, or trade secrets, it was really the right to privacy which was driving these outcomes. Cites to property rights were a fig leaf of precedent the courts were using to get to just decisions, but the decisions were felt to be just because someone's privacy, not property, rights were violated.

Having concluded that we already recognize privacy rights, the authors then try to define the actual boundaries of this right (e.g. "The right to privacy does not prohibit any publication of matter which is of public or general interest." id.), and suggest some remedies, drawn from existing causes of action for defamation. Essentially they try to make it easy for the courts to start to openly recognize this right by providing them with tests and limits to it so the judges don't have to do it for themselves (and can cite the esteemed Brandeis in doing so).

So that's the first significant legal push for a legally recognized right to privacy in the US. Two things to notice. First - the authors first talk about the positive outcomes having a right to privacy would lead to and only then suggest that such a right has been there all along, if hiding behind the euphemism of property rights. Second - the discussion doesn't mention or rely on the U.S. Constitution generally or, more specifically, its Fourth Amendment. Although the right of the people to be secure in their papers from unreasonable government search and seizure sure sounds like the right to keep stuff private from the government, the start of the argument for property rights was to found it in English common law almost exclusively.

This was not the case with the Supreme Court decisions which 1) formally recognized a right to privacy; and 2) grounded that right in its interpretation of the U.S. Constitution rather than in common law.

The first of these is the little-talked-about today but in retrospect critically consequential decision Pierce v. Society of Sisters, 268 U.S. 510 (1925), where the State of Oregon tried to mandate every student attend public schools with no exceptions, and attempted to prohibit private schooling. The Society of Sisters of the Holy Names of Jesus and Mary was a private religious school that challenged the law on various grounds including that the law interfered with its property rights (there they are again) because its grounds were only suitable to be used as a school so outlawing that use amounted to taking their property (also outlawing the contracts between parents and school amounted to interference with right to contract). The court began by saying that, as a corporation, the school didn't get constitutional due process protections for its contracts if they were impaired by proper government action. But, looking to property law cases, the Court concluded that the Oregon law was not proper in that it interfered with freedoms of families and school.

The reason this case is such a huge deal is that this was the very first time the Supreme Court recognized 1) 14th amendment due process protected individual civil liberties; and 2) the first time the court explicitly (instead of in dicta) recognized constitutional rights applying to corporations.

The seminal case to truly cement privacy as a Constitutional right was Griswold v. Connecticut, 381 U.S. 479 (1965), which held a Connecticut law criminalizing contraception illegal for violating the Constitutional Right to Privacy. The Court famously found this right in the 'penumbrae' of the amendments to the Constitution (1, 3, 4, 5, and 9, specifically). In doing so it cited past cases where Constitutional rights were held to have penumbrae. In Griswold the Court discussed how, for example, inherent to the freedom of association is the right to be private in who you associate with, how inherent in the 4th amendment's protection is the right to keep your papers private, etc. It concluded that making contraception illegal violated the right to "marital privacy." It was about 75 years from the first article about a right to privacy until it was central to a major Supreme Court holding. Whether you consider that long or not, it doesn't seem to have been much of a battle - it was a right that was slowly recognized by the courts rather through a big public debate. The consequences and extent of that right to privacy after it was recognized is where the real debate and battle appears to have been, with it underpinning Roe v. Wade (and progeny) and Lawrence v. Texas (holding anti-sodomy laws unconstitutional).