Perhaps the fact that Alexander Hamilton used his influence with the electors and created propaganda campaigns against certain candidates in order assure an outcome favorable to the Federalist Party in the first 3 - maybe 4 presidential elections.

There is a painting of the first Governor of New Jersey and also was the Governor of New York, Edward Hyde/Lord Cornbury and it is of him in "drag" or in other words, he is dressed as a female. It has been speculated that this was a slander propaganda against him, and another theory posed is that he actually practiced cross dressing. It's important to note that there is no supportive evidence of Lord Cornbury cross dressing, aside from this portrait.

Aside from this, he was corrupt too and he was eventually arrested for unpaid debts. When he was arrested he was no longer liked and had a bad reputation regarding his sexuality.¹

¹http://nymag.com/news/features/scandals/edward-hyde-2012-4/