Hooray! A question I can answer. :D I'm a PhD student studying twentieth century animal history and would love to eventually write a book about cats in American history.

So, the "crazy cat lady" trope has been around for a very long time. In Colonial America, cats were often viewed as pets kept by witches or a potential animal form for the Devil to disguise himself. Superstition helped fuel these beliefs, and the cat is not off to a good start.* Also, cats were associated with unmarried women or spinsters. These women were considered a burden on society and nuisances. However, much of the "crazy cat lady" trope and cats being associated with females can be traced back to female sexuality. In her book, Beast in the Boudoir: Petkeeping in Nineteenth Century Paris, historian Kathleen Kete explains how the cats were intertwined with female sexuality. The cat was mysterious, wild, and sensual - much like a woman. The cat desired love and sexuality, but was also dangerous. Again, this returns back to Christianity and the tempting to tame women's sexuality. Cats were also thought to be more independent and attached to one person rather than a large group of people, or even indifferent to their owners. Even today, society thinks of cats as more of an independent and one person animal vs the dog which is family oriented.

In the 19th century, as historians Harriet Ritvo and Katherine Grier, in addition to Kathleen Kete, have shown -- the cat becomes gradually more accepted as a family pet. We begin to see the development of purebred cats such as the Angora, Persian, and Siamese among others. Cats begin to be bred for size, color, and coat variation rather than function like dogs. The Cat Fancy is created in the late 1800s and the first cat show is held in 1871 in England. The first American cat show was held in 1878.

In the 20th century when women were fighting for the right to vote, cats appeared in anti-suffrage advertisements. Why? Association of the cat and the suffragettes told citizens that only bitter, angry, unmarried or old women wanted to vote. However, cats were still primarily kept as mousers rather than companionship. The domestic pet cat like we are familiar with today does not begin to gain popularity until the post World War II era. People realize that cats are great for apartments and urban living. (Kitty litter becoming readily available in addition to scratching posts and commercial cat food also helps)

EDIT: So, the “man with cat” trope is interesting because historically, it is not the case. Yes, cats were thought of as being associated with females but no one looked down on men having cats. In fact, the first president of the Cat Fancy Association in England was a guy. Samuel Clemens aka Mark Twain had cats. Men and women participated in cat shows, so it was not a sign of being unmanly for men to have cats.

Sources:

Kathleen Kete, Beast in the Boudoir: Petkeeping in Nineteenth Century Paris

Katherine C. Grier, Pets in America: A History

Harriet Ritvo, The Animal Estate: The English and Other Cultures in the Victorian Age

Susan D. Jones, Valuing Animals: Veterinarians and Their Patients in Modern America

Jennifer Mason, Civilized Creatures: Urban Animals, Sentimental Culture, and American Literature, 1850-1900

*If any historians of early America happen to see this post, and have sources that sources on cats in Colonial America -- you'll be my best friends. Virginia DeJohn Anderson's book, Creatures of Empire, though fantastic, barely mentions cats. I've been unable to really find any sources on cats in early America; so any help would be great.