There were many issues, one of the most important and one that you've already identified being the imposition of taxes after decades of very light governance and then almost as important as the initial imposition, the quick removal after some protest. This gave the Americans the belief they could successfully challenge the government. Much of the later protest, political agitation and conflict sprung directly from opposition to taxation and the belief it could (and should) be challenged.

However one that you haven't identified yet is that the removal of the French threat to the North and West, which removed one of the forces that unified the colonists and British government. It also opened up a huge swathe of land that the Americans wanted to get settling as soon as possible, to hell with the Native Americans that were already on the land. The British government was not against this but wanted to protect the Natives to a certain extent and also to manage the process and divide the land amongst it's favourites, who could then sell it on. The Americans were somewhat annoyed by this to say the least, especially those already living on said land in places like the future Kentucky and Tennessee.

This influenced the creation of the Proclamation Line, which was always meant as a temporary line designed slowly to shift westwards and manage the expansion of the colonies over a matter of years, however the Patriot press very successfully propagandised this as a permanent ban on Westward expansion and of favouring Natives over Englishmen. There was uproar against this, turning a good deal of frontiersmen against the Crown..a few years later these same frontiersmen would deal the British an important defeat at the Battle of King's Mountain. People like George Washington were also very active in western land speculation and this soured their view against the government.

Later in 1774 Parliament also passed the Quebec Act which allocated parts of these new lands around the Ohio to Quebec and allowed the French Canadiens all kinds of religious and legal rights which thoroughly outraged the WASPy types in the colonies, both for the religious tolerance of former Catholic enemies and because it seemed to further restrict settlement in the new lands.

These issues combined with the taxation/representation issues and other movements and began the spiral to unrest and protest. Then the British government's reaction to that civil disobedience was cackhanded at best, resulting in the Intolerable Acts, and further fuelling the protest, igniting it into a rebellion.

The British government was also utterly incompetent at cultivating it's own support base in the colonies and supporting those that were pro-crown. This lead to all the colonies falling to Patriot control in the years leading up to outbreak of the Revolution and the suppression of the roughly one third of people who remained loyal to the crown and the quiescence of the third who remained neutral. So while many accounts of the Revolution may make it sound as though there was this seismic change across the whole nation to an anti-British position, it was actually far more nuanced with one prominent well supported faction mobilising more efficiently and driving events. This is not to say that there weren't massive changes, some might even say revolutionary changes (ha!) occurring throughout the colonies and their political views but merely that it wasn't like the whole of the North American continent had turned against the Crown, even as late as 1774. There were plenty of people living in the colonies either ambivalent to the revolution or passionately pro-crown, for them the events of the inter-war years hadn't change their core position at all.

Divided Loyalties: How the American Revolution Came to New York by Richard Ketchum is a great book for understanding the inter-war years through a New York perspective. It follows two families, one Loyalist (DeLancey) and one Patriot (Livingston) and how they lived through the inter war years, the political tumult, how they came to come to their positions and the roles of all the various factions. Although it's a New York account and furthermore an account of two landed families rather than common citizens, there's much in common with how the rest of the colonies were dealing with the situation and forming their opinions and how a certain faction of colonists turned against the British Crown and Parliament and drove the Revolution.

Historians argue that Americans intended reform British rule and eventually came to the realization that independence would be the only solution. Your question assumes that American opinion turned over the space of two decades, but in fact, the timing of the turn to revolutionary sentiments is one of the hot debates in the historiography right now. Much of the historiography argues that the break comes in the mid 1770s—not as a two-decade long process.

Pauline Maier’s book From Resistance to Revolution argues that through the 1760s, Americans hoped they could reform the government by appealing first to the parliament then to the king. American colonists along with English and Irish radicals shared. John Wilkes, a anti-ministerial satirist, libertine, and advocate for British liberties with an infamously ugly mug, became a transatlantic symbol for governmental reforms during the era. Distrust of the British political system, particularly the ministry, stands as an important factor in the building discontent. However, Maier argues it was only with the publication of Common Sense that independence became the imperative.

Brendan McConville’s book The Rise and Fall of Royal America emphasizes the degree to which colonials drew closer to the imperial center over the course of the eighteenth century through print, celebrations of the monarchy, political culture, holidays historical consciousness, sermons, genealogy, etc. Drawing on older work, he argues that American colonists held onto an older constitutional understanding in which sovereignty rested with the Crown, while in Britain, political thought following the Glorious Revolution evolved separately toward greater emphasis on the Crown-in-Parliament. These breaks began to emerge in the 1760s, but the colonists did not blame George III until about 1773-4.

Thomas Slaughter’s recent book Independence brings together the argument for a late turn toward revolution with the traditional narratives of salutary neglect and the colonial drift away from Britain. The colonial experience on the frontier, the War of Austrian Succession, the Seven Years’ War, the Great Awakening, etc., all contributed to colonial divergence. Americans developed a spirit of independence and a reputation for lawlessness. Colonials increasingly wished to be independent within the empire, but the British misinterpreted it as a desire to leave the empire entirely and took increasingly firm measures to keep them subjugated. He argues that Americans did not want independence until British troops took up arms against colonials in the Battle of Breed's Hill/Bunker Hill in June 1775.

I would stress though, that even during the Revolution, as the other comment began to say, about 40% of Americans were patriots, another 40% were neutral, and 20% were staunch loyalists. Even by 1776, Revolutionary sentiments had not even reached a majority view.