Curious: how would any answers to this be sourced to the standards of r/askhistorians? Any authoritative documentation of Deng's intents and strategies would almost certainly remain classified information in the custody of an intelligence organization.

Thatcher herself later said that during the negotiations between the two sides that Deng had told her that "I could walk in and take the whole lot this afternoon" and she had replied that "there is nothing I could do to stop you but the world would now know what China is like". Lu Ping, the chief Chinese negotiator during the handover process later confirm that Deng was not bluffing and China would have invaded not only if the talks failed but also if there had being a popular uprising in Hong Kong which prevented the handover from taking place.

Hong Kong was geographically indefensible and both sides knew it. Hence why Hong Kong had a token garrison which was incapable of defending against a serious Chinese attack. The official US report NSC 5717 in 1957 evaluated the value of Hong Kong as a propaganda and intelligence gathering asset but concluded that the colony was not defensible and hence the US would not render combat assistance to the UK in the event of a Chinese attack. Throughout the cold war era the HK garrison was continuously reduced and the policy was to deter rather than defend against a possible Chinese attack. After the 1967 riots demonstrated the vulnerability of the British position there there was no serious discussion of the British garrison as anything other than an internal security force.

As an aside China wouldn't even have needed to invade to get the British to fold, all it had to do was cut off the food and the water.

Source: Eastern Fortress: A Military History of Hong Kong, 1840–1970 By Kwong Chi Man, Tsoi Yiu Lun

Underground Front: The Chinese Communist Party in Hong Kong By Christine Loh

There's actually a similar thread from a while ago that has a pretty good response that at least in part answers your question. I would like to add some input though.

Just before the negotiations in 1982, Deng Xiaoping adopted a new unification plan involving Taiwan, Macau, and Hong Kong. He wanted to show that he could compromise and negotiate in order to persuade Taiwan to rejoin China. Deng planned to use the "one country, two systems" model, originally meant for Taiwan, on Hong Kong as an example to Taiwan. Hong Kong would become a Special Administrative Region (SAR) after reunification and would keep a high degree of autonomy with little involvement from the central government. This plan persisted through the negotiations and made it into the Joint Declaration, which allowed Hong Kong to be a SAR for 50 years after the handover.

Given this information, it would be highly unlikely that Deng would want to forcefully invade Hong Kong. Taking the colony by force would not be a good way to persuade Taiwan to peacefully rejoin China. Moreover, as China's economy opened up, it wanted to benefit economically from Hong Kong. Forcefully taking Hong Kong would have at the very least undermined the public's confidence and thrown the economy into chaos, which happened several times leading up to the handover, such as after the Tiananmen Square Incident.

However, going along with /u/DeSoulis' comment, Percy Craddock, who was the British ambassador to China during the first part of the negotiations, claimed that Deng Xiaoping was determined to recover Hong Kong fully, "even if it meant taking it back as a barren rock." Though this is the closest we can probably get to Deng's intent during the meetings now, because these quotes are coming from a British perspective, I'm inclined to believe they were slightly exaggerated and viewed through a slightly different cultural lens. As much as Britain wanted to let the people of Hong Kong live under British rule as they wished, China saw the recovery of Hong Kong as a way to end the unequal treaties of the Opium Wars and maintain face and honor. This would in part explain China's persistence to recover Hong Kong during the negotiations. And also as /u/Desoulis has pointed out, Britain had no bargaining power as both sides knew how indefensible Hong Kong was, so China could afford be persistent.

Though there is no documentation (that I know of) of Deng's intent, based on the historical context of the time, it's pretty safe to assume his general thoughts. Unfortunately, I can't site directly my first paragraph since I'm getting it from secondary sources, which site those passages in Chinese (which I can't quite read yet). So I can't say exactly how historians know of the creation "one country, two systems" plan, but these secondary sources both have similar accounts regarding this point and are pretty reliable.

Sources:

Carroll, John. A Concise History of Hong Kong. Lanham: Rowman & Littlefield Publishers, Inc., 2007.

Tsang, Steve. A Modern History of Hong Kong. New York: I.B. Tauris & Co Ltd., 2004.