John Adams was a bit ill-tempered as a man, which worked against him. In addition, he had the same curse that any VP succeeding the prior president has: he couldn't help but be compared unfavorably to George Washington. Toward the end of his presidency, he also got rather paranoid that Hamilton was exerting too much control over his cabinet (I've read Chernow's Hamilton, the basis for what you listened to, but I've got no idea about the musical, so no clue if this is addressed in there). He could kinda be described as a President who felt it was 'owed' to him as repayment for his work on behalf of America throughout his lifetime.

However, in addition to that, there are a good amount of reasons that he was frowned on that have more to do with his policies than his personality. The most prominent of these are the Alien and Sedition Acts (https://en.wikipedia.org/wiki/Alien_and_Sedition_Acts), which were both hard on immigrants, extending the length of time for naturalization from 4 to 15 years and putting really tight constraints on freedom of speech, including getting rid of the right to criticize the government.

They were incredibly unpopular and aroused complaints of dictatorial and tyrannical powers. Even his VP, Jefferson, undermined him. (At the time, candidates did not run on a ticket like, say, Obama-Biden or Bush-Cheney today; the top vote getter won the top position, the runner-up became VP. This could lead to some pretty awkward situations, and Adams-Jefferson is effectively the epitome of this, though some credit has to go to Jefferson-Burr, as Burr killed Hamilton, fled, and then was involved in a conspiracy, allegedly, to create an independent country in the center of the US; he was arrested for treason, though acquitted). If a president were to try and implement similar policies today, I wouldn't be surprised if you had an immediate widespread move for impeachment, honestly, particularly for the harsh restrictions on freedom of speech.

It also alienated and galvanized his opponents, so they were overtly political moves. He was a member of the Federalist party and generally, immigrants tended to vote for the Democratic-Republicans. I hate to draw in modern day comparisons, especially one that can arouse really strong emotions, but it'd be a bit like a Republican president making extensive efforts to chop down immigration in order to cut down the potential voter base of the Democratic party.

The XYZ Affair, where the French demanded exorbitant bribes before being willing to speak to American diplomats really lowered opinions on the French, but also reflected poorly on Adams. Though he ultimately succeeded in preventing a war, these are the two big things that reflect poorly on Adams, and they also played a pivotal role in preventing him from getting a second term.

He was also a bit of a dick: he got the hell out of dodge in the middle of the night after Jefferson won the election rather than attending Jefferson's inauguration, a snub that is incredibly rare in US history, though, admittedly, precedent at that time had not yet been sent.

Since I haven't seen the musical myself, this is borderline moronic of me to say, but I imagine you might enjoy Chernow's book, which is phenomenal, if it is a bit of a doorstopper. He addresses these issues really well. If you're more interested in Adams, however, I'd advise David McCullough's Adams biography. I have yet to read it myself, which is why I haven't cited it, but it's sitting in my reading stack and has generally been more highly regarded and well-known than Ferling's, the Adams biography I cited.

Sources:

John Ferling, John Adams

Ron Chernow, Hamilton