So, as you mentioned, after the Opium Wars, Hong Kong Island (1842) and Kowloon (1860) were ceded to the UK, and in 1898 the UK leased the New Territories for a period of 99 years. China at the time was very divided, and remained so through until the creation of the PRC, with the fall of the Qing empire, the Warlords era, encroachment by the Japanese and Russians and the Chinese Civil war all making China a very weak power until unification under the PRC in 1949. The UK at the time was at the height of the empire, and remained strong until essentially exhausting its capabilities in the Second World War.

The Christmas 1941 occupation of Hong Kong in little more than a week by the Japanese proved that Hong Kong was no longer defensible by the UK using standard warfare, and the UK's defensive budget only shrank in the post-war period. Holding out against a now-united China militarily was considered a possibility in the early 1950s, but planning relied upon the support of the US, possibly even the use of the US nuclear arsenal. The US were understandably cautious about creating a 'Berlin in the East', and refused to give the UK a firm commitment to guarantee Hong Kong's security. Though the debate went on for several years, by the early 1960s it was clear that the only military option was planning a rear-guard action to provide enough time for the evacuation of essential personnel from the Colony. In other words, by the mid-1960s the Hong Kong government and FO officials in London admitted internally that the Colony only existed at China's sufferance.

There were further issues beyond the purely military as well. Though the ceded territory and the leased territory were supposedly different, in practice this distinction was not maintained from the start. land suitable for building was at a premium around Kowloon and there was a rapid expansion from the ceded territory into the leased territory, creating what became New Kowloon, contiguous with 'Old' Kowloon. In the event of the leased areas being returned, there was no realistic place a border control could be imposed. This was only the start of the infrastructure issues, without giving an exhaustive list, more or less all of the colony's domestic food production was in the New Territories, all but one of the colonies major reservoirs, the airport was in New Kowloon, and the New Towns initiative, designed to reduce infrastructure stress on Kowloon and Hong Kong island meant that an ever larger percentage of the population lived in the New Territories. In sum, the ceded area could no longer function if a hostile or even unfriendly China had taken over the New Territories.

By 1970, this had become clear to the Foreign and Commonwealth Office who presented a paper to Cabinet with the options for the future of Hong Kong towards and after 1997. Apart from the possibility of a hostile takeover of the colony, they foresaw several options.

a) Do nothing - Some thought that China was happy with the arrangement that existed, because of the economic benefits that Hong Kong brought to China (particularly as a hole in the economic embargo on China), and that if the topic was not brought up, then nothing need be done to force the Chinese to state their position. The lease would be allowed to expire and British rule would continue.

b) Unilateral withdrawal - After the collapse in civil order in Macau (just across the bay from Hong Kong) in 1966, the Chinese had gained an effective veto over the administration of the Portuguese territory. If this were to happen it would have been very embarrassing to the British. Thus if this seemed likely then a unilateral withdrawal might be better than an ongoing humiliation of British power.

c) Negotiated withdrawal either before or in 1997 - If the Chinese gave indications that they could be trusted to negotiate in good faith (relations at this point being quite hostile), and were adamant about recovering Hong Kong, then a bargain might be struck for an orderly transfer of sovereignty.

d) Negotiated extension of the lease - Hong Kong was after all a prosperous place and, as noted above, Hong Kong was an asset to them. Thus they might be willing to continue the present arrangement beyond 1997 in exchange for an ongoing rent from the government of Hong Kong. Even if the price was high, the Chinese population and expat and foreign business community might be willing to pay to keep the security of the UK's rule of law and their way of life unchanged.

Thus even at this stage it can be seen that there was no real thought of maintaining the ceded territories as a separate entity.

1971-2 would prove a crucial year for the future of Hong Kong with two important events. The first of these was a conversation between a former British parliamentarian and diplomat Malcolm MacDonald and Zhou Enlai in the course of the negotiations that led to full diplomatic relations between the UK and China. Zhou (though through backchannels) gave MacDonald an assurance that there would be no hostile takeover of Hong Kong, and subsequent discussions over the following year expanded on this to say that Mao and Zhou had no intentions that China should reclaim Hong Kong until the end of the lease in 1997. This then ruled out options a) and b) above, although some distrust remained on the part of the British, as the Chinese would attempt various diplomatic efforts to weaken British sovereignty over the colony through the 1970s. The second important event was that in 1972 at the UN Hong Kong and Macau were removed from the list of dependent territories shortly after the Chinese entered the organisation (and why this came to pass is something I'm working on at the moment). This effectively made Hong Kong's future a bilateral matter between the UK and China, and removed the last possibility for an independent Hong Kong.

So, as you can see, the possibilities for Hong Kong's future had become very limited by the early 1970s. Either negotiated withdrawal or negotiated extension, and no real possibility of self-determination or independence.

When Margaret Thatcher went to Peking in 1982, she at first tried to suggest the latter idea to Deng Xiaoping, by then undisputed leader of China, as the best option for the continuing economic utility of Hong Kong to China. There were some indications that he might accept this deal. Deng had been the protege of Zhou and ended the hardline Maoist rule of the Gang of Four and begun the process of economic reform of China, proving he was a pragmatist. However, in this meeting Deng said that he would rather recover a destroyed Hong Kong than allow a functional Hong Kong to remain under British control. As to why he insisted on this, my opinion is that, without the legacy of Mao or Zhou to fall back on as a great national hero (at least this early in his rule), and after disavowing Mao (at least partially), he needed a alternative source of legitimization. Rather than an abstract political ideology, he chose a much older political tool and turned towards a more traditional nationalism, hoping to reunite Taiwan, Hong Kong and Macau with China, and undo the last 'humiliations' of the 19th century.

Thatcher held out against concessions for around a year after this initial meeting, but on the advice of the UK Ambassador to China, Percy Cradock (who would later become one of her closest aides) and the Governor of Hong Kong (Teddy Youde) she finally accepted that the best possibility was to negotiate the best possible 'special status' for Hong Kong under Chinese Sovereignty. This led to the Joint Declaration of 1984, which specified that Hong Kong would become a Special Administrative Region which retained much of the British legal system and economic model, though with final arbitration from Beijing. The UK got the 'best deal possible' under circumstances where China held almost every card.

This is very much the 'orthodox' version of the history and my opinions differ in some areas, but as yet I can't prove everything, or link it up so I'll leave this here for the moment. Ask again in a year.

Sources:

Secondary

Chang, D. W. & Chuang, R. Y., The Politics of Hong Kong's Reversion to China, (Basingstoke, Macmillan, 1998)

Cheung, G. K., Hong Kong's Watershed: The 1967 Riots, (Hong Kong, Hong Kong University Press, 2009)

Cottrell, R., The End of Hong Kong: The Secret Diplomacy of Imperial Retreat, (London, John Murray, 1993)

Hong Kong Research Project, Hong Kong: A Case to Answer, (Nottingham, Spokesman Books, 1974)

Mark, C., Hong Kong and the Cold War: Anglo-American Relations 1949-1957, (Oxford Oxford University Press, 2004)

McGurn, W., Perfidious Albion: The Abandonment of Hong Kong 1997, (London, National Book Network, 1992)

Patrikeeff, F., Mouldering Pearl: Hong Kong at the Crossroads, (London, George Phillip, 1989)

Roberti, M., The Fall of Hong Kong (2nd Ed.), (Chichester, John Wiley & Sons, 1996)

Tsang, S., A Modern History of Hong Kong , (London, I. B. Tauris, 2004)

Welsh, F., A History of Hong Kong (2nd. Ed.), (London, HarperCollins, 1997)

Wesley-Smith, P., Unequal Treaty: China, Great Britain and Hong Kong's New Territories 1898-1997, (Oxford, Oxford University Press, 1980)

Yahuda, M., Hong Kong: China's Challenge, (London, Routledge, 1996)

Primary

National Archives, UK:

FCO 40/265

FCO 21/808-815

FCO 58/681

PREM 19/789-790

PREM 19/1053-1059