While /u/funkyedwardgibbon has looked very instructively at why Britain could not defend Hong Kong, I’ll examine why they would not - why they did not want to.

Already in March 1969, a British government study concluded that "Hong Kong’s future must eventually be in China", that the British government’s "objective must be to attempt to negotiate its return, at a favourable opportunity, on the best terms obtainable for its people and for our material interests there."

The triggering event for the study had been the Maoist riots of 1967. Essentially, these came out of Chinese leftist students adopting the mainland Cultural Revolution mindset and trying to adopt it to struggle against the British colonial overlords – only the Chinese government did not want that to happen, more on that later. So a great mess occurred, tourism and economy suffered, the international press reported, but nothing came of it in the end because most of the population was totally alienated by the Maoists – apart from the British questioning their own role in Hong Kong.

To quote the then-British foreign minister: "We have to accept the fact that China could take Hong Kong at any time of her choosing either as a result of a direct military attack or by making our position in the colony untenable through organised internal uprisings. Therefore our aim should be to gain time in the hope that circumstances may be such in the future that agreement with China over the return of Hong Kong might become a possibility."

Why Britain did not want Hong Kong

As /u/funkyedwardgibbon has pointed out, Hong Kong would have been indefensible had the Chinese decided on a military offensive. Still, the British role as colonial overlord demanded for a military presence that the British were conscious could never be enough to actually defend the city, only to uphold public order; this meant that Hong Kong authorities were dissatisfied with having to battle the British for increased defence spending while the British had to spend millions on defences they knew would be inadequate.

This actually represented the majority of their spending on the city. Otherwise, there was little spending – and little control. As Mark sums it up, “geographical remoteness, lack of Parliamentary interest, shrinking resources within Whitehall, and the healthy financial position of Hong Kong” all contributed to a policy of reducing direct British control as much as possible.

At the same time, and on a note that seems highly ironic today, Britain was increasingly orienting itself toward Europe during this time. The European powers did not want Hong Kong to enter the European common market for various reasons, the Commonwealth office was merged into the Foreign Office which was more concerned other matters – as Mark writes, "the Crown Colony became an increasingly embarrassing political anachronism" while "Europe and China were both crucial to Britain’s future in an emerging multi-polar world."

Why the Chinese were happy to let things run

Why were the Chinese happy to let Hong Kong stay in British hands until 1997? Because a British Hong Kong presented them with vast economic opportunities desperately needed to develop mainland China. In 1966 alone, the Chinese had ~200 million pounds of foreign exchange earnings through Hong Kong.

The Chinese position had been basically unchanged since 1949: First, they did not recognise the treaties that brought Hong Kong under British rule; second, they would still do nothing against British rule; third, they would not accept an independent Hong Kong inside the Commonwealth, which was considered as a serious possibility by British officials.

This was summed up by Zhou Enlai during the negotiations for the establishment of regular diplomatic recognition of China by Great Britain, when he said that China had "no intention of seeking to get Hong Kong back until the expiry of the New Territories lease."

Conclusion

These were the basic positions of Britain and China. There were of course also the inhabitants of the city – Chinese, British, Hong Kongers or anything in between – and it’s important to remember that they also had agency for themselves and were not simply a playing ball between a waning Empire and a rising superpower. But it’s a topic that deserves to be covered in full, and your question was on why the British let Hong Kong go, so I’ll leave it here.

I’ll also leave the question of how much Hong Kong was "democratised" up to 1997, which is interconnected with the previous one. It’s interesting to note that here China did in fact take quite a hard stance on British and Hong Kong plans, but again, that’s a big topic in spitting distance from the twenty-year rule, so I’ll leave it be.

So, to sum up: The British did not want to spend money on Hong Kong, they were not all that interested in Hong Kong, it was a nuisance to closer integration into Europe and closer relations with China. At the same time, they were happy to keep it on a long leash if the Chinese let them – and the Chinese did, because it gave them economic opportunities and because time was working in their favour.

Sources:

Mark, Chi-Kwan: "Development without Decolonisation? Hong Kong’s Future and Relations with Britain and China, 1967-1972", in: Journal of the Royal Asiatic Society 3 (24), pp. 315-335.

Tsang, Steve (ed.): Government and Politics. A Documentary History of Hong Kong, Hong Kong 1995.

Essentially: if China attacked, Britain couldn't defend it. If China didn't attack, Britain couldn't keep the water running and the lights on. It was untenable to hold it past 1997.

To expand:

Britain first took territory in Hong Kong in 1841, when Hong Kong Island was ceded to the British Government by the Qing Dynasty during the First Opium War. That was a preliminary agreement made with a Qing viceroy in Southern China, and the war continued even after Britain took possession of the island, but it was ratified in 1843.

Hong Kong possesses a fine harbor, and it was a very useful Royal Navy base for securing British interests in the South China Sea, and later in the interior rivers of South China. The population rapidly grew, and soon Britain was looking to expand onto the nearby Kowloon Peninsular- this was good flat ground suitable for both farming and the training of the garrison. It took Kowloon during the Second Opium War in 1860. This was part of what's called the First Convention of Peking.

Now, it's important to note that this territory was not leased. It was ceded to the British in perpetuity. Legally, the Qing government and its various successors had no claim to them (in law. Morally and in terms of nationalism, perhaps a different story.)

In 1898, however, the British expand north yet again. Now, by the late 1890s the diplomatic situation has changed somewhat. The Qing are weaker than they have been for decades, especially after their humiliating defeat in the First Sino-Japanese War. Whereas even in the 1880s the Qing had the capacity to stand up to foreign powers and resist territorial encroachment (as with Russia in the Ili Valley in 1881,) there's a general sense now that the Manchu court can't protect itself.

China was seen as an incredibly important emerging market, and influence there was key to any Western power's empire in Asia. However, unlike in Africa, there was also a reluctance to formally carve up the country. Now, whether or not that would have been possible in practice is debatable, but the point is that it was not tried. Britain and America in particular wanted to keep the Chinese market open to everyone, as their traders were probably the most competitive, and no one else wanted to risk a confrontation.

So instead, the empires leased coastal territory instead. These were small enclaves, designed to be refueling stations for western navies and useful bases to spread their influence inland. So in March 1898 Russia took a 25 year lease of the area around Port Arthur in south-western Manchuria (this would later be the chief area of contention in the 1905 Russo-Japanese War.)

Germany and France, almost simultaneously got 99 year leases of territory in the Shandong Peninsular and Zhanjiang (near Hainan island) respectively.

The UK, fatefully, signed the Second Convention of Peking in June of 1898 and secured a ninety nine year lease on what's now called 'The New Territories.' This vastly expanded Hong Kong's area- it not only moved the British-China border north, it also leased smaller islands in the surrounding waters such as Lantau. It hugely increased the arable land in the colony, making the great population boom of the twentieth century possible.

But it was, in the end, a lease. Now, legal historians will be better able to explain the importance of a 'ninety nine year lease.' In my understanding, at the time it was basically understood to be a lease in perpetuity- the longest possible lease in civil law. The British negotiators weren't foolish enough to think it wouldn't run out; they just didn't imagine that Britain would ever be weak enough relative to China that it couldn't simply renew the lease.

They did not imagine, in other words, the People's Republic of China.

I'm a historian of the 1890s, not the 1980s, so I cannot speak to the details of the Thatcher Government's decision that it would seek to negotiate a handover of all Hong Kong in 1997 rather than trying to hold on to the permanently ceded territories. I can, however, point to the facts that would have informed the decision:

In 1997, the PRC (which Britain had recognised as the legitimate government of China and the legal successor of the Qing since the 1950s,) was going to take possession of the New Territories. Their legal case was indisputable.
They were not guaranteed to acquire southern Kowloon or Hong Kong Island itself.

However, the New Territories included:

86% of the city's territory. Half the population. Most of the city's water supplies. Most of the city's power supply. The airport.

And, not least, it had been proven in WW2 that the city was completely indefensible by land.

So even if Britain had opted to hold on to its original holdings, and had the PRC for some reason not simply occupied the city- which it was more than capable of doing- Britain could no longer have kept the city's basic civic infrastructure functioning without PRC cooperation, which would not have been forthcoming.

The Falklands and Gibraltar are not leased, and both can be supplied locally even in the face of hostile neighbors. Even in the worst case scenarios, Spain and Argentina never had anything like the military capacity necessary to challenge Britain. The great challenge facing the UK in the Falklands was the supply lines to the South Atlantic, not the Argentinian military. Holding Hong Kong in the face of a vastly superior Chinese force that was also a nuclear power, on the other hand, was completely untenable.

TL;DR: They had neither the legal grounds to keep Hong Kong, or the capacity to keep it running, or the military might to defend it.

Further reading:

Julia Lovell's 'The Opium War' has good material on the cession of Hong Kong itself. Diana Preston's 'The Boxer Rebellion' has, in the sections on the background of that conflict, a good report of the various European encroachments on China including the lease of the New Territories. For a general history of Hong Kong, John Carroll's 'A Concise History of Hong Kong' is good, but if you want something that gives a better sense of late British Hong Kong try the travel writer Jan Morris's 'Hong Kong.'