I thought I would add a little to the already excellent answer given by /u/mimicofmodes. I think it would be interesting to look a little at the history of class, especially the way class is represented and talked about. (I'm afraid I'm going to speak in pretty broad terms, since this is a huge topic.) The "gentleman" in Victorian society has roots in the landed gentry from previous centuries, who owned land and made an income from tenant farmers, but did not have titles themselves. The term "gentleman" dates back to hundreds of years before the Victorian period, but its use would shift significantly. Around the turn of the 19th century (early 1800s), two major social changes were occurring which would combine (among others) to form what we see today as the Victorian gentleman. On the one hand, there was the industrial revolution, which greatly expanded access to wealth. A man not from a wealthy family could make quite a lot of money from investing (railways, foreign mines, and shipping could all be solid money-makers), from banking, from owning a factory, or from professions like law, medicine, science, academia, and literature. (But as Chris Rock told us, just because you're rich doesn't mean you're wealthy... we'll get to that.)

The other change was a shift in public perception. It was traditional that the aristocracy had the claim to being better people: they (believed they were) were simply better behaved than the unwashed masses, who dirtied themselves with immoral things like labor and wanting money (it should be noted that having money was not seen as immoral--and it was, let's be honest, not something the lower classes could be accused of anyway). Courtly behavior and chivalry could be expected of the aristocracy, but the common people (the churls) would act... well, churlishly. Even today, the word "vulgar," deriving from the Latin "vulgus" for common person, refers to things that are in bad taste. But by around the late 18th century, that was changing, led in large part by a change in how people saw the aristocracy. And there was no better place to point the finger than the Prince of Wales/Prince Regent/King George IV, with his shockingly wasteful parties, his famous appetite, his alcoholism, and his nearly constant scandals meant public opinion of the nobility was at an all-time low (not to mention Republican sentiment wafting over from France during and after the Revolution). In contrast to this was the new middle class's moral claims to authority: unlike the aristocracy, they were (or rather claimed to be) modest, well-behaved, and respectable. They also sought to change the way work was seen. A profession (not something dirty like working with your hands, but good decent work, like being a lawyer or insurer) was a good thing. It showed that you were industrious, disciplined, and that you were contributing to society. These ideas might seem natural to us now, but these ideas represented a significant shift.

As /u/mimicofmodes points out, this is one of the reasons etiquette was such big business in the 19th century: it was possible to be rich without being a gentleman (though being a gentleman without being rich was much more difficult!), and one had to have the manners to go with the wealth. Taste also became very important: it was not enough to just buy art, but one had to demonstrate that they had good taste by doing so (displaying bad taste was something like having plastic flamingos on your lawn!). Even some popular music, such as Richard Wagner, could be considered bad taste: more conservative people said it would inflame the passion of young people, and should be avoided! (Even today, we often say that someone with good taste and manners has good breeding: even though one could be a gentleman without having those old family connections, the terminology remained entrenched.)

One of Charles Dickens's biggest interests in his writing was in the gentleman, and we see some of the contradictions and pitfalls of this new social construction in his writing--especially since part of the implied impetus in his writing was to establish that people like him were gentlemen. And central to being a gentleman was respectability. That form of that word is really important: it wasn't just that someone had to behave well, but they had to be able to be respected: your reputation, your appearance, your behavior all mattered, so that people could see that you were respectable--though of course trying to show that you were respectable was also a no-no (like trying to be cool in high school). A perfect example is from Dickens's novel David Copperfield: born to an indebted family, young David's hands are literally blackened from work in a factory as he works to get his father out of debtor's prison. If anyone knew of this shame, it would be his social ruin. (It's worth mentioning that Dickens himself had similar work as a boy, for similar reasons, and that he kept it secret from almost everyone.) As he works his way up, David lodges with a respectable lawyer and meets his daughter, Agnes. David aspires to marry her, since having a respectable wife was a great asset towards being respectable, but this is difficult, because marrying for class or wealth would make him an adventurer (in the bad sense of someone trying to get rich, not in the Dungeons and Dragons sense).

In a distinctly Victorian contradiction, it's good to work your way up, but it's bad to be ambitious. Working hard is good, accepting advancement as a result of it is good, but actually wanting that advancement (and being seen to want it) is bad. Instead, David marries Dora, who calls herself his "child-wife": she represents many of the ideals of womanhood (she is sweet, affectionate, and innocent), but she is a bad match for David, because she cannot keep a household or help him in his work: in other words, she would have made a good aristocrat, but she's no good as a middle class wife. David, meanwhile, is working his way up as a writer, first writing for court papers, and then aspiring to be a novelist. After Dora conveniently dies after childbirth, David, now financially established and with a good reputation, can marry Agnes without having to worry about how it will seem: he has the financial means now, and he has behaved (more or less) like a gentleman. (He did end up breaking the heart of Em'ly, a low-class girl David was in love with as a boy, because she just wouldn't have been the worthy wife of a gentleman.)

The novel David Copperfield has (at least) two characters who are like evil mirror universe versions of David, and therefore show what a gentleman should not be like. Uriah Heep, the legal clerk who keeps going on about how "umble" (humble) he is and that he doesn't want to get above his station, nevertheless wants to marry Agnes. He ends up being caught out for his illegal and immoral business practices and sent to prison. The other mirror I wanted to mention is Steerforth, a former schoolmate of David, who is arrogant and snobbish. He is charming and comes from a well-to-do family, but (like Heep) lacks moral character: he seduces Em'ly with no intention of trying to make her his (respectable) wife and abandons her, and ends up dying in a storm. So that was a long tangent, but I hope it illustrated some of the important parts of what a gentleman was (and wasn't): he was diligent, morally upright, respectable in the eyes of society, and lacking in (visible) ambition. His wealth is (represented as) a result of his talent and honesty: after all, a gentleman does not seek his own gain....

(I should also emphasize that not all novelists were gentleman. In fact, most weren't: if being a gentleman had been a given, Dickens wouldn't have had to devote so much paper to the question! Often, the things that are written about/talked about the most are right at the boundary, because the obvious extremes don't even enter into the question, and the "are they or are they not?" is an endless source of discussion!)

As a fun note before I end: Charles Dickens, annoyed by Americans publishing his novels in pirated copies, is said to have remarked, "I do not know the American gentleman, God forgive me for putting two such words together." Some Americans certainly were very rich. At least according to Dickens in the heat of the moment, though, they lack some other qualities that would make them gentleman...

I've already talked (er, written?) your ear off, so I won't talk too much (more) about changes, but the history of the (British) gentleman is very much tied up in political reform, so check out the history of the Reform Acts of 1832, 1867, and 1884 if you'd like to see some of how society was transforming legally as well as in practice. In many ways, it is a sign of the times that Benjamin Disraeli, a novelist of Jewish descent and the second son of a scholar, went on to become one of the most famous Prime Ministers of Queen Victoria. But we can talk about him another time. :)

Sources, you say? Given my background, I love The Idea of the Gentleman in the Victorian Novel by Robin Gilmour, and it was a major source for my PhD work. I also like Lauren Goodland's Victorian Literature and the Victorian State: Character and Governance in a Liberal Society. They're old sources and could well be outdated (please tell me if they are!) but I have also gotten a lot out of Eric Hobsbawm's massive three-volume history of "the long 19th century," The Age of Revolution: Europe 1789–1848, The Age of Capital: 1848–1875 and The Age of Empire: 1875–1914.

At the most basic level, what made a man a gentleman was simply that he didn't work for his money. He had enough capital that it could be invested in the government funds, and the four or five percent dividends it brought could sustain him, as well as his family if he had one, possibly supplemented by rents from tenants on any land/in any houses he owned or sales of minerals mined on his property, etc.

However, there's another angle. For one thing, younger sons of gentlemen who had minimal inheritances to lean on needed to do something to support themselves - typically they went into the church (which would eventually give them a secure living, as discussed at length in this past answer), or the law (which would often lead to a seat in Parliament), or the army (their father could purchase them a commission as an officer). Despite earning money, they were from "good families" and their social connections were all to the upper and upper-middle class, and so were still considered gentlemen - besides, their occupations didn't really have employers or set hours of work, they weren't relying on sales, and their hands rarely got dirty. (Physicians might be considered gentlemen if their fees were high enough for similar reasons.) And at the same time, men who earned a fortune in trade as merchants of some kind, then invested it and lived off the proceeds the same way acknowledged gentlemen did ... might still be looked-down-upon by the more established gentry as an invader.

Such men and their families often didn't know the unwritten rules of high society, which led to the proliferation of etiquette books and conduct literature in the nineteenth century. Texts like The Gentlemen's Book of Etiquette, and Manual of Politeness (1874), Good Manners (1888), A Manual of Dignities, Privilege, and Precedence (1844), and The Laws of Etiquette (1836) were eagerly purchased by the upwardly mobile in order to learn the rules that they hadn't been able to pick up in childhood and adolescence like their new peers. If you're looking for the social rules of being a gentlemen, books like these are enormously helpful.

You might be interested in my past answers on Victorian high society, particularly Looking at Charles Booth's Poverty Map of Victorian London, what sort of occupations, if any, did the upper-middle class hold? and Why did the ultra wealthy start having nominal occupations instead of just being aristocrats?