This one is rather outside my area of specailization, but I'd liketo take a shot at it anyway.

Much of the English aristocracy in the 18th century were more or less constantly in debt. There were some structural reasons for this. One was the nature of incomes for the landed gentry. While they made some money from the rents, the vast majority of their wealth was in the form of rural real estate while is obviously a highly illiquid asset.

At the same time, performing their indentities as members of the gentry and "gentlment of leasuire" required constant expenditures of conspicous consumption. This might be in the form of building big expensive manor houses, financing electoral interests, or sometimes just simple gambling. The result of all this was that in the 18th and 19th century, aristocrats generally morgaged their properties heavily and took on enourous debt loads to finance their conspcious consumption. As long as they held title to real assets, aristocratic families could usually find a banker willing to continue to loan them money, even if the value of the debts exceeded the nominal value of the deeds used as collatoral by several times.

Despite this, aristrocratic bankruptcies seem to have been fairly rare occurrance, because in an age of falling or static interest rates, land ownership provided what was seen as an nearly iron-clad gauranetee to secure mortgages and annuities. In fact part of the problem is that prospective heirs would take out "annuity loans" that they would promise to pay once their windfall inheritance materialized, so that they could live the high life of a libertine in the present. (I am editorializing somewhat here, perhaps.)

Probably the best book length-treament of this subject is John Habakkuk's massive economic history of the English upper classes Marriage, debt, and the estates system: English landownership 1650–1950.

For a more in-depth look at a case study of a upper-class familiy protypical of kind OP finds often in literature of the period, I suggest "The Finances of the Dukes of Chandos: Aristocratic Inheritance, Marriage, and Debt in Eighteenth-Century England" by Dickinson and Beckett (2001).

I confess I have not read either of these in a great while and honestly haven't kept up with the developments in this area of scholarship over the past decade or so, my scholarly interests having shifted to parts further east and south.

Sources:

Dickson, P. G. M., and Beckett J. V. "The Finances of the Dukes of Chandos: Aristocratic Inheritance, Marriage, and Debt in Eighteenth-Century England." Huntington Library Quarterly 64, no. 3/4 (2001): 309-55.

John Habakkuk, Marriage, debt, and the estates system: English landownership 1650–1950 (Oxford: Clarendon Press, 1994).

I can't speak for the 18th, but for the 19th and early 20th century (1800-1914) I have a little bit of knowledge as to why this might be a common theme in literature that's centered around that time period.

As Western Europe (in this case focusing primarily on Great Britain, France, and Belgium) industrialized, international banking became more prominent. As the economies of these industrialized countries became more dynamically complex, governments began to shift away from the prevailing-economic ideas of the colonial days (mercantilism) and instead began pushing for free trade capitalism.

It's a lot more complicated then that, but the general gist is that as Great Britain industrialized and began lowering trade restrictions with other nations, immense wealth started flowing back into the country. Much of it was in private hands (as in, not part of the government and instead owned by citizens.) As a result, Great Britain slowly became the industrial and financial capital of the world.

Now, for the part that's relevant to your question: many of these British 'gentlemen' of the time began to export their wealth (or, their capital) to fund the expansion of industrialization in other countries. It's not a stretch to assume that these ventures didn't always end up being profitable, and that the British 'gentlemen' of the day had to borrow money to get back on their feet, or to further fund their costly industrial ventures in foreign countries.

Source: Global Capitalism, It's Fall and Rise in the Twentieth Century

A point to note: Most gentlemen at the time had their wealth in land; their income was ultimately from farming. Farming has the characteristic that the income is all at harvest time, in the fall, but the expenditure is across the whole year; so farmers (then and now) often borrow against the harvest just as a usual means of doing business. Then if the harvest is bad, you have a problem. Notice that this is also a common theme in literature of the Great Depression and the American settlement more generally: You have to raise $X by the time the mortgage is due or you'll lose the farm. Incidentally, this is also why banks offer mortgages with yearly installments, which tends to seem a bit crazy to people with biweekly or monthly paychecks. :)

Another point is that wealth in land is not very liquid, and the English squirearchy hated selling it anyway; a gentleman who needed a lot of ready cash might prefer to borrow with the land as security rather than wait for the harvest to come in.