160 years ago the UK ran a rather primitive form of education. Public schools (fee paying) existed for the richest but most others received limited religious instruction, focusing on reading the Commandments and little else. It wasn't until the late 1800s that elementary schools (providing more comprehensive and compulsory education in the three Rs) became a thing. Grammar schools as we recognise them, that is passing an academic entrance exam, were formed under the 1944 Education Act.

In a hypothetical situation where a working class individual could have gotten into Oxbridge and entered the world of politics there would still be a social stigma attached to their position. Even the new industrialist class of the 1800s were relatively frowned upon. Furthermore, an MP would need some form of financial support - until the 1832 Reform Act 95% of the British population could not vote and votes were reserved for the wealthy, propertied classes. MPs were also not paid. Morally, the wealthiest saw the poor as beneath them, uneducated and lacking in airs and graces.

(I hope this is helpful - I'm unfortunately on my phone so editing and references are a bit lax.)