Thanks for asking this question!

First things first, 18th and 19th century academics did not necessarily have the same understandings of hominid classification as we do now. It was entirely possible for British academics and anthropologists writing about Australian Indigenous peoples to consider them obviously human but also to be an inherently inferior and ancient type of human. Over the course of the 19th century there was an increase in 'racial' science and writing that analysed, categorised and ranked human populations by intelligence, worth and morals - with Europeans, of course, being considered the pinnacle. This type of 'academic' literature was written about many groups - Africans and East Asian people were also regularly 'discussed' in these terms - but Indigenous Australians, in particular, were focused upon and dehumanised.

Whilst not going so far as to scientifically classify Indigenous Australians as entirely outside humanity, there was effectively consensus amongst Western racial scientists that Indigenous Australians were uniquely the lowest living form of human beings. Out of Indigenous Australians, Indigenous Tasmanians and other 'Tasmanoid' Indigenous people were further singled out as the 'lowest of the lowest'. Indigenous Tasmanians, famously, suffered ethnic cleansing, war and genocide - and literature describing them as subhuman was often both a preceding and retroactive intellectual justification for these events. Racist ideas from this period, such as the idea that Tasmanians didn't know how to use fire, survived well into the 20th century, and even the present. Keith Windschuttle, a denialist, repeated the fire claim in his 2001 book on the Black War.

Historians in Australia will use the word 'polygenistic' (yes, it is a mouthfull) to talk about Australian scientific racism. Within 19th century Australia, and the UK, it became common to assert or argue that not all modern humans had a common origin - and that in particular Indigenous Australians were clearly not of the same human origin as Europeans. There were a handful of academics who argued that Indigenous Australians did not mature out of mental childhood - and were perpetually children. In this regard, yes, many 19th century white Australians did not regard Indigenous Australians to be human in the same sense as they were.

Racism against Indigenous Australians almost always focused on Indigenous societal 'primitiveness' - lack of settled agriculture, 'nomadism' - and bitter feelings from a history of brutal violence and frontier conflict. The spectre of the night-time raid or massacre of white settlers by Indigenous warriors was a regular touchstone in Australian discourse about race. Dehumanisation is a common part of war, and wars where massacres and ethnic cleansings were common especially so. Racial categorisation and 'science' allowed European Australians to retroactively justify violence against Indigenous peoples as morally okay and as the only course of action possible.