The classic book on the topic is Sydney Mintz's Sweetness and Power, written a few decades back. It's a great read, if a bit dense. (Mintz is actually an anthropologist).

Short answer: it's complicated.

Long answer: Mintz offers five chapters, including one on Food Culture (sugar was originally a preservative, and a status symbol of the aristocracy), Production (sugar production used slavery and proto-industrial methods to make sugar cheap), Consumption (the cheaper sugar got, the more people could consume it, and the more they demanded it, in a sort of feedback loop), Power (sugar's role in social class roles and the factory system), etc.

It's really hard to distill all of his great arguments, which include the evolution of culture and taste...

So here's one cool side-point: coffee & tea are both caffeinated, and sugar is caloric. When you put sugar in tea, it reduces the bitterness, allowing it to be drunk more quickly. As the modern industrial system emerged, factory workers needed:

-a break (though employers needed those breaks short--no more than 10 minutes)

-energy

-calories

So tea with sugar became the perfect addition to the modern factory system: you could drink it quickly (because of the sugar), got a bump from the caffein (and the sugar itself), got calories to continue... it was the perfect trifecta. Then it worked its way into English culture, and from there became linked to identity.

Any source for the claim that "the Muslims" didn't use sugar either? In Iran at least almost all tea is drank with sugar. Anyone have any info on the history of sugar in West and South Asia?

Follow-up question: When and why did milk start getting added to tea?