There was not much there on the island of Hong Kong when the British took it over in 1841.

There were said to be 7,450 fishermen and charcoal burners living on the island.

http://en.wikipedia.org/wiki/Hong_Kong

The Emperors of China in the 1660s had ordered the "Great Clearance", which removed all people from the coastal areas of Guandong, to limit smuggling, piracy, and contact with foreigners.

The area had not really recovered much population before the British took it over.

Adding to the above: there are evidences of significant human activity in Hong Kong before the Great Clearance that's reflected in archaeological discoveries and place names. This topic is not fully understood, and frankly even the Hong Kong Museum of History doesn't say much about human settlements in Hong Kong before the Brits came along, other than just showing some artifacts and say "they existed" without much local context.

Back in the 1960s a Han dynasty tomb now known as the Lee Cheng Uk Han Tomb was found in the Cheung Sha Wan area. It was spacious enough that whoever was buried in it had to have been an influential man. Such a man and such a tomb here suggests that Hong Kong had been inhabited to a great degree already, 2000 years ago.

The area of Kwun Tong in East Kowloon was named after the official salt fields there established in the Northern Song period. Kowloon itself was graced by the presence of the Southern Song emperors fleeing from the Mongols in the 13th century. This was how Kowloon got its name, by the way, since the name means "nine dragons" - the dragons being the eight mountains surrounding the peninsula and the emperor himself. That the imperial entourage stayed here for some time gives credence to the theory that Kowloon may have had some significant settlement during the Song dynasty. In fact, subway construction crews have dug up village ruins in the last few months in the To Kwa Wan area, where we already have a large stone commemorating the Song emperor passed down from the Mongol Yuan dynasty!

The area of Tuen Mun got its name from the fact that it was the sea gate that was garrisoned with soldiers. During the Ming dynasty in the 16th century, the first Portuguese explorers to China actually landed here, where they recorded its name as "Tamao" and erected a stele of discovery. The Chinese authorities saw this and their fortifications as an infringement of Chinese territory and expelled them in two battles collectively known as the Battles of Tamao: one in Deep Bay where the Hong Kong-Shenzhen border is now, and another probably near Castle Peak. The battles are now trumpeted in Chinese museums as the "first clash between China and Europe where we defeated the first imperialist colonizers". Hong Kong's own museums are silent about this episode.

Then, in the 17th century, we come to the Great Clearance event. The Manchu Qing took the drastic measure of evacuating everyone within 30 miles of the coastline and razed everything inside in order to isolate the Ming loyalist regime in Taiwan. Needless to say, this is the reason why Lord Palmerston saw Hong Kong as nothing but a barren rock, and also why pre-British Hong Kong history is so muddled and out of context nowadays.