First, there were plenty of "non-elites" who initially did not follow the elites into revolt, even ones who did not think of themselves as Loyalists. In North Carolina before the War there had been something of a revolt in the western mountains against the exploitation of the planter-dominated government in the east. The "Regulators" were brutally suppressed, and when that planter-dominated government wanted to fight against the British, the western settlers were reluctant to join. Similarly , most of the Hudson Valley of New York was owned by several wealthy families, like the Livingstons, and most of the farmers there were tenants. There was a lot of anger - justifiable- among those tenant farmers towards their landlords, and they likewise were quite lukewarm about fighting alongside them.

There were also groups that saw their best chances to be with the British. Virginia Governor Lord Dunmore offered to shelter any slaves who would leave their rebellious masters, and many enslaved Blacks would escape and seek shelter behind British lines. Many of the Native Nations saw an alliance with the British , or at least neutrality, as being much preferable to allying with the Patriots. The Colonies had shown themselves to be very hungry to take Native territory.

Then there were all the people who simply hoped things would be patched up. Like Quaker John Dickinson of Philadelphia, who dispatched a last petition for peaceful reconciliation in 1775, not knowing that the British government had already set in motion a military effort.

But there had been high-handed treatment of the colonists by the British government for quite some time before the famous Stamp Act and Townsend Act. The colonies had very much been seen primarily as a benefit to Britain, and to that purpose colonists had been restricted as to who they could trade with, and had been barred from setting up their own manufacture of some goods. Those restrictions had been constantly circumvented, and were sometimes repealed, but were still there in theory ( for example, a ban on making hats- beaver pelts had to come from the colonies to Britain , made into hats, and then returned for sale to the Colonists). There was also the Proclamation of 1763, which ended the hopes of the colonists for the taking of Native territory that was no longer under French protection. The Proclamation gave the British government the exclusive right to negotiate and gain lands from the Native Nations beyond the Alleghenies, removing a lot of territory from settlement. This was a very unpopular move. The colonies had an agricultural economy, and land was , essentially, money. To be denied land was to be denied opportunity, advancement.

But there was also a general feeling that the colonies were already running themselves, could run themselves. That the British government and society, with its dukes and kings, thousands of miles and a month's voyage away, was either hindrance or useless. This was well-articulated in Thomas Paine's Common Sense. Paine's pamphlet seems to have been instrumental in clarifying the colonial move to independence, and many joined the movement because of it. Of course, it should have been obvious that the British government had been useful, sometimes. The colonists own efforts to fight the French and Indian War had stalled until a large British professional army arrived. But that was easily forgotten- or excused.

But all these existing grievances would perhaps not have been enough if the British had also not begun to make war in the colonies. In suppressing the revolt, the British created more and more resentment. In the early stage, in 1774, there had been a riot in Boston when the British moved their own gunpowder from the city armory, the famous Powder Alarm. That was a real popular uprising by an already fearful, restive population- it was not led by any Boston Brahmins. But it was only small event, and the British hoped to limit the rebellion to the Boston area, or at least to Massachusetts. But instead , their military action had pushed a lot of people who'd been vacillating into opposition. The revolt grew in size, and by October 1780 even the settlers in western North Carolina, who had previously had no love for the NC government, rose up against the British in the Battle of King's Mountain.

So, TL:DR There was already some popular resentment towards the British government, but as the war continued that grew to be a greater and greater majority.

Paine, Thomas. (1894) The Writings of Thomas Paine: 1774-1779. Putnam's.