#1/2

This is a complicated topic and I don't believe it actually has a very simple, clear answer. Most obviously, the change from touching and outward affection between two men being a common aspect of male friendship to it being indicative of a romantic or sexual relationship is tied to the changing ideas about homosexuality in the late 19th century. But it's also connected to ideas of "racial degeneracy" posited by race scientists like Arthur Gobineau and his antecedents, which fueled a particularly bellicose style of nationalism around the turn of the 20th century, and to a changing American relationship to sports, expressed not only through a more professionalized culture of promotion and consumption, but also in how sports were organized and played on college campuses. I should say straight out that this answer is drawn primarily from sources that focus on the American expression of this change in masculinity, and I can't speak with any confidence about similar changes in other cultures.

Masculine Affection in the early 19th century

The OP has already brought up at least one example of the kinds of mutual affection men were comfortable publicly expressing, but I think it's worth talking about a few more. The very close relationship between Alexander Hamilton and his friend John Laurens left us with letters between the two that openly and fervently expressed love:

Cold in my professions, warm in my friendships, I wish, my Dear Laurens, it might be in my power, by action rather than words, to convince you that I love you. I shall only tell you that ’till you bade us Adieu, I hardly knew the value you had taught my heart to set upon you.

This can (and has) been read as evidence of a sexual relationship between the two men, but it's also a very common element in many public friendships between men in the late 18th and early 19th centuries. Love was understood as something that encompassed far, far more than mere sexual desire, and expressing love and fondness, and using effusive poetic descriptions of your affection to another man, was just a thing men could do.

And because we will return to this, there was a concurrent comfort in the descriptions of men and men's bodies that was taken as appreciative and non-sexual. William Webb, a town father of Hays City, Kansas, described town marshal James Butler Hickok as having "the shoulders of Hercules and the waist of a girl." Many men - and women, including Libbie Custer - made a point to describe his hair and his sky-blue eyes. A certain physical femininity in a man was seen as attractive, desirable, and even powerful before (very roughly) the 1880s or so.

So, to restate the question in the OP, how did this acceptance of non-sexual love and tenderness between two men become the kind of public homophobia common in the late 20th century?

Homosexuality

Anthony Rotundo's American Manhood describes in brief the cultural shift in the idea of homosexuality following the US Civil War. He describes two interrelated trends:

the growth of areas in major cities where those interested in sexual relations with members of their own sex could meet and develop a sense of community: and a distinct change in the way society viewed that sexual behavior

In essence, while homosexuality was believed to be "deviant" or "unnatural" and was for the most part explicitly illegal, it was also somewhat quiet; there were very few prosecutions for the criminalized acts of same-sex love in American cities be prior to the Civil War. This is partially because queer communities were small and remained necessarily hidden. By the 1880s, those communities felt safe enough to make more public expressions of their lifestyle, and subsequently became more visible. This led to a corresponding increase in prosecution and criminality, and for "scientists" and medical professionals to take an interest in the "problem" of homosexuality.

Older ideas in the United States about same-sex love and desire all assumed that it was unnatural, and that it was, as Rotundo explains, "an episodic visitation from without." The act was the socially problematic element, not the person. There were few other arguments about it, apart from criminalizing the acts as a matter of course. But by 1882 there was a new trend in (what amounted to) scientific thought: it was a degenerative disease. That is, it had natural causes that could be rationally explained. This shifted the focus from the act to the person.

Science, or something like it

19th century science was nothing if not enthusiastic. Following the industrial revolution, the idea that science and rational experimentation had developed into ideas that it could also know the world. The world was rational, its expressions in the natural world could be known, and known, resources and wealth could be measured, extracted, and turned into profit. This pattern was the assumed purpose of the sciences, even if many individuals pursued learning for personal ideals rather than to assist in the domination of industrial capitalism. Nevertheless, the idea that science and rational inquiry could know the world soon extended into what we'd now call the social sciences as well. If science can understand how a bat could fly or plumb the secrets of electricity, it could also understand how cultures were constructed, and, as perhaps the most 19th century idea, how great cultures rose and - especially - how they fell.

Enter Arthur Gobineau. Gobineau was, like many others, interested in the reasons that great cultures or empires rose and fell. Like most others, he used ancient writings, especially those of the Romans, as a springboard for his analysis. The Roman cocktail for why nations failed was commonly repeated: irreligion, addiction to luxury, waning morality, and an increase in fanaticism and extremism were all a cultural "softening" that rotted the foundations of greatness, and led to their subsequent fall. This was the typical intellectual take for most scholars interested in the question, with emphasis on the religion aspect for some, on the fanaticism for others. But Gobineau didn't find this satisfying, because these "poisonous blossoms" (as he termed them) were self-evidently present in almost every society. So for a society to fall from prominence, there must be some "stronger principle of destruction."

To make a long story short, Gobineau's suggestion to this was race. Race and racial degeneracy was the principle that must be mixed in with the other poisonous blossoms to bring a great empire to its knees. We can follow the rather hideous ideology and its connection to ideas of homosexuality readily: race was a collection of essential, inborn, biological traits that led to measurable differences in physical, mental, and moral capabilities. Some races were smarter, some were hotter, some were calm, some were hotheaded. Whites, according to Gobineau (and just about every other race theorist), were the ideal mixture of these traits. Possessing innate superiority as a matter of biology simultaneously meant a couple of things: whites were the cause of the rapid course of progress traced over the last century, and whites would be responsible if it fell.

So because fanaticism, irreligion, amorality, and addiction to luxury was apparent in every culture without necessarily causing irreparable harm, the thing to isolate and control was race. And race theorists - and politicians, like the college-educated insatiable reader and amateur intellectual Theodore Roosevelt - proposed two ways to promote this control: legally limiting the possibility of race-mixing by banning or controlling interracial marriages or forcing sterilization on "lesser" races; and promoting ideal public manhood among the white citizenry.

Part of the promotion of ideal public manhood was promoting a kind of sexual self control or discipline, and that was immediately connected to the newer ideas of homosexuality. If homosexuality was inborn or biological, then control and prevention of homosexuality had to be policed at an individual level, too. The expression of this that we still unfortunately have in modern American culture is the increased use of personal slurs to describe gay men. Some of those slurs focused on the supposed effeminacy that was the assumed expression of homosexuality, and so, as the decades progressed, the visibility of outward signs of affection between male friends began to be viewed as evidence of the man's queerness. Even the physiognomy of a man might be taken to indicate that he wasn't straight.

These last couple of elements, the increased use of personal slurs and the idea that a man's physicality and robustness was indicative of their innate sexual desires, led to (or developed parallel to) changes in sport culture in the United States.