From my understanding, the taxes imposed on the colonists did have a significant economic effect on the colonists, but didn't plunge them into extreme poverty. It was perceived more as a problem because of the nature of the things being taxed. For example, the Stamp Act required that virtually any document, including newspapers, legal documents, and even playing cards, had to printed on specially embossed "stamped" paper produced and transported all the way from London. More inconveniently, the tax had to be paid in British currency rather than the paper money used by most of the colonists. An appropriate analogy may be that everyone was required to use a government-run internet network, and that they had to pay a tax in sterling silver or gold every single time they connected to the Wifi.

The Sugar Act was in many ways worse on the colonists, who were already experiencing an economic downturn. It continued the work of the earlier, less enforced Molasses Act by curtailing the use of sugar and molasses, which were crucial in the production of rum. The act also limited the export of certain commodities, notably lumber, so that the colonists could only export to Britain. The tax certainly hurt merchants, ports, and the business of rum production and rum smuggling, which had increased under the Molasses Act.

Certain people, especially merchants, were certainly negatively impacted by the taxes. But your question about whether it was for economic or ideological reasons that the taxes were protested can be answered by examining the rhetoric used by the protesters. A popular argument used by the colonists was about the nature of taxation without representation: it ignored their right to govern and tax themselves, or at least to have a representative in Parliament, where the taxation policies were being made. No one seemed to be arguing about the economic effects or asking for reduced taxation from the crown: it wasn't the amount of taxes that was the cause of complaint, but the act of taxation without representation itself.

A good example of this rhetoric is this quote from Samuel Adams: For if our Trade may be taxed why not our Lands? Why not the Produce of our Lands & every thing we possess or make use of? This we apprehend annihilates our Charter Right to govern & tax ourselves – It strikes our British Privileges, which as we have never forfeited them, we hold in common with our Fellow Subjects who are Natives of Britain: If Taxes are laid upon us in any shape without our having a legal Representation where they are laid, are we not reduced from the Character of free Subjects to the miserable State of tributary Slaves.

TL;DR: The economic impact of the taxes was certainly significant, but many of the protesters found the act of taxation itself, rather than the amount, to be the violation of their liberties. Of course, we must always consider that the rhetoric being utilized by the colonists was more ideological than truly motivational: it may well be that the colonists were economically motivated, and wanted to protest the taxes so they could set taxes (that they believed to be more fair) for themselves, and were using rhetoric of liberty and rights as a means to that end.