Thank you for this great question!

they were not displaced from their land

Sadly, this isn't true. In fact, the colonisation and conquest of Zenadth Kes, the name for the Torres Strait preferred by its Indigenous peoples, includes many of the kinds of violence and dispossession common in Australian history. If you're familiar enough with Australia's colonial past to be aware of things like the Black War, the ethnic cleansings in Central Queensland, Blackbirding or the Hawkesbury and Nepean wars - you'll understand that we're talking about a very violent experience of colonialism.

As an example, let's take the experiences of the Kaurareg people of the lower-west islands. In 1869, the cutter Sperwer was attacked while trawling off of Muralag island, in Kaurareg territory. It's eight-man crew was murdered, two of them white and five of them Malay, and the public in the Colony of Queensland flew into an uproar. As was standard practice in colonial Queensland - the response to Indigenous violence was to send a punitive expedition. This is what some historians might euphemistically call rough justice, but can be more accurately described as the sanctioned murder of Indigenous peoples.

Following the killing of the crew of the Sperwer, several raids were launched from Somerset in Cape York to Kaurareg territory. Frank Jardine, then Police Magistrate, lead the first one and was accompanied by several Native Police troopers, the ship Melanie and its armed crew, who were mostly "kanakas" - a term used at the time for labourers from many places in the South Pacific (such as Fiji or the Solomon islands). The records for the first raid are patchy, to say the least, but according to Jardine's son -

"The village was in a bit of a valley, high at one end. The kanakas burned it all out . . . They had rifles and no one was prepared. It was too sudden. There was a terrible killing"

It is uncertain how many died in this first raid. Upon returning, Frank Jardine took leave of absence as Police Magistrate, and was replaced in his duties by Henry Chester. Jardine's personal records of the raid have been lost, but all evidence we have that remains indicates that a massacre did take place, although some have noted that Jardine's son may have deliberately shifted blame onto the kanakas for the massacre.

Following reports that the wife and child of the Sperwer's captain were not only still alive, but stuck on the island, Henry Chester lead another raid. Accompanied by two troopers and nine indigenous men from Somerset, the raiding party took two Kaurareg men hostage and threatened they would not be seen again if the islanders did not release the captives. After twelve days, Chester decided that the reports of the captives being present on the island were most likely false - and released the two hostages.

But - a full eight months later Chester lead another raid in search of the captives. This time they had HMS Blanche - a frigate, twenty five bluejackets, eight Native Police troopers and eleven indigenous men from Somerset. They revisited the site of the massacre, took twenty local men prisoner, fired their camp and in the process of interrogation, several Gudang people identified three of the prisoners, who were Nagir men, as responsible for the Sperwer massacre. Chester had them shot.

To top it all off, the Captain's wife and son who had been reported as captives on Muralag hadn't actually been on the Sperwer. They were in Melbourne at the time of the voyage, and had remained there throughout the events just described.

This is one small story from the early days of colonialism in the region. It was believed that the massacres above had largely decimated the Kaurareg population, until survivors were discovered on Thursday Island in the 1880s. Throughout the early twentieth century, their population numbered under a hundred and at various points was removed and relocated. As late as 1922, Karaureg peoples were forced at gunpoint to relocate to Moa Island, where they remained until 1947.

In essence, the Indigenous peoples of the islands still had a negative experience following annexation - and the regular, and violent, intervention of outside government into their affairs often resulted in great suffering for many indigenous peoples. They still suffered the same punitive violence, and later paternalism, that Indigenous nations on the mainland did.