This article talks usefully about a fascinating opposition faction: Quaker pacifists. Quakerism as a religion prescribes very strict pacifism and the American history of conscientious objectors features them prominently, right back through the Revolution itself. Indeed, Philadelphia high society was a hotbed of at least reluctance about the Revolution, including non-Quakers, because of the long history of Quaker influence there (and the socio-economic prominence that Quakers had taken on in Philadelphia, even as they became a minority there). John Dickinson was a prominent opponent of Independence right up until it was a done deal and frequently spoke as such in the Continental Congress. He did, eventually, go along with the Revolution, but was not in favor of it.

Quakers are a special case of a broad tendency of what's called 'passive obedience'. This was especially common in certain parts of the Anglican Church in America. The distinction between loyal and patriot factions in the Anglican Church can get complex, but the simplified distinction I've read is between 'High Church' Anglicans, whose sympathies extended back to the Church Hierarchy of the Mother country and would have welcomed an American bishop, and 'Low Church' Anglicans, who theoretically supported an episcopal church polity but were used to running their own religious affairs through their local vestries. While there is an element of self-interest you can read into 'passive obedience' rhetoric, I'm the kind of person with more willingness to believe that people believe what they say, so there was an ideo-religious aspect to preaching 'passive obedience' amongst Anglican churchmen who did that. While this isn't an area I have much expertise in, I believe the language pre-dates the revolutionary era and has roots in actual religious doctrine, so there's something to it not just being self-interest.

Other opposition factions included recent Scottish immigrants (who had taken loyalty oaths to the Crown they took seriously), back- and up-country farmers in the Carolinas who opposed the Patriot-leaning Tidewater planters for localized political reasons (see: The War of Regulation), slaves owned by Patriot planters up and down the southern Atlantic coast (who did have political agency, and split between those who hoped that service in the Revolution would win them their freedom and those who believed that they might win their freedom by joining the British against their revolutionary masters), and Scottish factors in the Upper South who were commercial agents for British firms that handled the purchase and export of Virginia and North Carolina tobacco. The Cousin's Wars by Kevin Phillips, while not having these groups as part of its central thesis, touches well on these various other opposition groups (plus more besides) and their reasons for opposing the Revolution, which varied from religious, ethnic, or based in local politics and recent local history. It's quite a good book, but be warned that the American Revolution is just one part of it (it includes a short section on the English Civil War and a much longer one on the American Civil War -- its central thesis is tying these three conflicts together, which it does an OK job on but isn't really why I recommend the book. Phillips is not a historian, although he writes a decent history).

Another, more recent, more scholarly-academic book on the development of the discussion prior to the Revolution is 1774, by Mary Beth Norton. It talks about the development of the Revolutionary movement in general, but the value of it to your question is it also talks about opposition, both to the broader movement of resistance to Parliamentary authority as well as to the increasingly radicalization of the resistance movement (it was interesting to learn about how widespread shock at the Boston Tea Party was). It's extremely well researched and does a good job of building a coherent narrative, although it does involve some familiarity with the events of the era already to be able to easily follow. If you want to purchase it, I recommend the hardback, even if it's more expensive, just because of the quality of the construction of it.