The thing is about the British Empire is it didn't meet just one end - that is to say, the pursuit of wealth for instance. It may have started out as such, with British America and then British India being solely founded on ambitions of world trade, but over the decades and centuries the remit of the empire changed considerably to the point where the powers-that-be reckoned the purpose of empire was to act as a paternal figure to all its subjects and basically hold the hand of its less "civilised" territories; by the 20th century they'd made so much money off of it they'd more or less stopped caring about financial gain. Part of the reason it floundered about and gradually collapsed after the Second World War is because people either lost interest in the empire or couldn't work out what to do with it. I think to answer whether or not it benefitted the lower classes you first have to pinpoint which period of the empire you refer to, as the British Empire circa 1750 was a considerably different enterprise to the Empire in say, the 1930s.

There are several arguments you should explore that suggest that the empire really benefited a small group (or class). These include Cain and Hopkins "Gentlemanly Capitalism" argument set out in British Imperialism 1688-2000 as well as a number of articles, and the careful study of imperial stocks & bonds in Davis and Hattenback's Mammon and the Pursuit of Empire. Both of these are loosely attached to a Marxian (or at least materialist) analysis that goes back as far as Lenin's 1917 study of imperialism and a liberal tradition dating to Hobson's 1902 Imperialism: A Study. Cain and Hopkins is nicely critiqued in the collected study edited by Ray Dumett entitled Gentlemanly Capitalism and British Imperialism.

One aspect of it was that it provided work for many average Britons. The Empire provided a huge market for British made goods, particularly textiles, and so provided a lot of income for factories manufacturing those goods. Of course they often were not paid well, but like any industrialising nation the working classes did see incomes significantly rise over the course of decades and centuries.

I'm writing this mostly from memory, but if you want a good book on this then I highly recommend Empire by Niall Ferguson.

To summarise, England's (later Great Britain's) colonisation of India and the West Indies increased the quality of life of all Britons astronomically. First of all, it must be said that the initial purpose of the colonies was purely economic. Originally, British (and Portuguese) possessions in India consisted only of fortified trading posts along the coast. These trading posts were controlled by companies, such as the East India trading company, which had monopolies over trade in a certain area. With the establishment of these trading posts goods that were previously considered to be luxuries, such as tea and cotton, began to be shipped to Britain in large quantities. Over time, as the colonies expanded, more and more resources were brought back to Britain, and eventually even common people could afford things like tea and good quality clothing.

By the mid 1700s the quality of life in Britain was significantly higher than, for example, France. Then, in the mid 18th century, the process of industrialisation began in Britain and things really took off. Certain factories enabled goods to be produced at 100x the rate they were before, and Britain also had a guaranteed market for its finished goods. The colonies would ship cheap raw materials to to Britain, and then Britain would sell them expensive finished materials. By this point, even poor British households could afford items that would have been restricted to the very wealthy only decades beforehand.

All in all the British people, including both the upper and lower classes, benefited greatly from colonialism. For much of the 18-19th century British people had a higher quality of life than most of the world, matched only by the other colonial powers such as the Dutch and Portuguese.

Edit: Upon rereading OP's question, I would like to make my answer a tad more specific. The expansion of the British Empire was very much for the benefit of the elite, but the influx of goods increased the quality of the lower class as well. This is essentially a perfect example of trickle-down-economics.

Establishing preferential markets and expanding the tax base were certainly justifications for empire, under an economic theory known as Mercantilism. However, in contrast to Marxists who view Imperialism as a kind of hyperdevelopment of capitalism, capitalist economists and historians (notably beginning with Adam Smith) have argued that empire didn't actually benefit anyone.

The practice of granting monopolies to well-connected companies and promoting exports while discouraging imports may have supported some economic development in the short run, but in the long run they were disastrous. Without competition, companies don't have to worry about product quality or availability or any of the other things that satisfy customers -- you know how this works if you've ever dealt with a company like Comcast only to find, when you try to cancel, that you have no other options for service in your area.

The British East India Company held a virtual monopoly on many trade goods, but it was frequently on the verge of financial collapse. Because its directors were well-connected, Parliament was always attempting to bail them out with one scheme or another -- such as the Tea Act that resulted in the Boston Tea Party.

One of the reasons it had continual financial troubles was because empire is expensive: paying, equipping, supplying and transporting troops and naval forces is expensive. In the case of governments, the government would pay for war with increased borrowing and by devaluing currency and then they would have to pay off the debt with more devaluation and rising taxes, the combination of which generally produced an economic downturn, especially for the losers.

Laws restricting imports also hindered the British economy. In the case of the American colonies, it allowed a number of people to get rich through smuggling. In case of Britain itself, the Corn Laws escalated the price of food dramatically, resulting in several riots and fears of revolution.

In short, empire really didn't benefit anyone. Colonization, like in Canada, the Thirteen Colonies, Australia and New Zealand, were more beneficial for the British than conquest, but still expensive to develop.