In 1842, when the British won the First Opium War, they had China over a barrel and could set their own terms for surrender. One of the terms of the Treaty of Nanking was that Hong Kong would be ceded in perpetuity. The whole reason Hong Kong mattered at all was as a natural deep draft harbour, and the treaty reflected this - only Hong Kong Island itself was mentioned.
In 1858, realizing that their needs were growing, Britain leased Kowloon up to Boundary Street, and in 1860 the Convention of Peking made the lease permanent.
As Hong Kong grew, so did the need for land - and meanwhile the British position of power had weakened. in 1898, Britain leased the New Territories for 99 years. Dealing from a slightly stronger position meant China was able to get another small concession - a small military outpost - a fortified village, really - was excluded from the lease.
From 1899 to 1945, the KWC was almost empty and treated no differently from any other part of Hong Kong, but in 1945 China re-asserted its rights to it. Squatters and refugees pored in, and after a brief confusion the infamous "anarchist enclave" was established. China was nominally in control, but generally only cared about the KWC for political gain and negotiation.
As the end of the 99-year lease drew closer, Britain and China entered into negotiations - starting in 1982 - to see what would happen when it came to an end. By 1985 it was clear that the end of the lease would mean Hong Kong was returned to China - which made the KWC unnecessary as a negotiating chip. Between 1987 and 1992, the 33 000 residents of the KWC were relocated, and in 1993 it was demolished - replaced by a memorial park.
Sources:
While not answering your question, /u/DubiousMerchant had a great response on another thread about Kowloon Walled City and referenced some source material that may answer your question.