I can’t speak to the question about the book itself and if it generated controversy, I suspect it may have but only slightly as a contribution to what was at the time a non-existent field to academia essentially (that being post-colonial social history). Counter-intuitively however the reason that the question of invasion is contentious now when it was not before is precisely because of the post-colonial social history that has been done in the decades since.

During the period of colonial expansion and in its immediate aftermath the archives are full of references and lamentations of the “invasion of Australia”. Multiple conflicts with Aboriginal people are referred to as “wars”, there was substantial humanitarian efforts, particularly in the colony of Victoria to try and stem the tide of destruction that was apparent both to settler governments and peoples but to people in England and around the world. Then, after federation occurred in 1901 this essentially stops. At the time and throughout the first half of the 20th century, Australia had in place a policy of segregation, control and “breeding out” of Aboriginal people, who were denied citizenship and voting rights and who were controlled by state protectorate boards in every aspect of their lives. In this framework, “Terra Nullius” as a legal fiction becomes law of the land. Officially, on the Federal level and according to most academics at the time, there was no one there before 1788. In the 1960’s an Anthropologist named Bill Stanner who was doing research on Indigenous peoples noted this conspicuous absence in the historical record and in academic fields, calling it “the great Australian silence”. In the 1970’s and 80’s historians like Henry Reynolds (the other side of the frontier) and Richard Broome (Aboriginal Australians) began writing histories of Aboriginal experiences of colonisation, and this field has expanded greatly over the years. However there was backlash, conservative historians disagreed with the attempts to mar the history of the country with what they felt was an exaggeration of negative events, and the derisive label of “black armband historians” was then picked up and used by John Howard both before and after he was elected Prime Minister. Australia is essentially still undergoing that culture war, and saying that the country was “invaded” signals which side of the war you are on, which when Hancock wrote the book, was not really the case.

Essentially this book came out at a time when Aboriginal history was completely overlooked and ignored, and I suspect the book itself may have been as well (I never came across it while doing my degree in Aboriginal history). But in the time since it was written the aboriginal history of the country has become a larger and larger focus of academic study and cultural discussion, and as such has been swept into the culture war, making it contentious. Hope this is at least useful even if doesn’t answer your question directly.

Firstly, to provide some context (taken from a previous answer of mine), we need to talk about the Great Australian Silence.

The term 'The Great Australian Silence' was originally the title of one of the anthropologist W. E. H. Stanner's 1968 Boyer Lectures. The Boyer Lectures was broadcast on the national broadcaster ABC radio at the time, and Stanner's series of Lectures were soon published in text form (and later in a collected series of essays, which I have, and which the quotes below come from). Stanner's lectures were about Aboriginal people, and in the lecture 'The Great Australian Silence', Stanner turned his eye to the way that Australian histories portrayed Aboriginal people...or didn't.

By 'The Great Australian Silence' Stanner means the systematic (and very convenient) removal of Aboriginal people from the grand narrative of Australian history typically discussed in history books. That is, Australian history talks about European colonists seeking to ...colonise, to settle on a particular piece of land and use it for their own purposes, without talking about the process by which that settlement must have happened (i.e., the removal of the people who had previously been the traditional owners of the lands). And especially without talking about the indigenous peoples whose lives continued after they were dispossessed of their lands. To quote Stanner, as he surveys some mid-20th century histories of Australia:

The next was George Caiger’s The Australian Way of Life (1953), in which the word ‘aboriginal’ is not to be found; no, I am wrong; it does occur— once, in a caption under a photograph which displays two of Australia’s scenic attractions, the Aborigines and Coogee Beach. To the next book, W.V. Aughterson’s Taking Stock: Aspects of Mid-Century Life in Australia (1953), there were ten contributors. Only one of them, Alan McCulloch, the art critic, has anything to say about the Aborigines, some passing but perceptive observations on their art. Incidentally, the book opens with a chapter entitled 'The Australian Way of Life’, written by W. E. H. Stanner, who can safely be presumed never to have heard of the Aborigines, because he does not refer to them and even maintains that Australia has ‘no racial divisions like America’.

Take that, W. E. H. Stanner! I wonder if they're related?

Anyway, W. E. H. Stanner's conclusion about why this is, is that:

A partial survey is enough to let me make the point that inattention on such a scale cannot possibly be explained by absent-mindedness. It is a structural matter, a view from a window which has been carefully placed to exclude a whole quadrant of the landscape. What may well have begun as a simple forgetting of other possible views turned under habit and over time into something like a cult of forgetfulness practised on a national scale. We have been able for so long to disremember the Aborigines that we are now hard put to keep them in mind even when we most want to do so.

And part of the reason for this Great Australian Silence according to Stanner, is that it asks some very awkward questions of people of European heritage, that they do not want to think about:

All land in Australia is held in consequence of an assumption so large, grand and remote from actuality that it had best be called royal, which is exactly what it was. The continent at occupation was held to be disposable because it was assumed to be ‘waste and desert’. The truth was that identifiable Aboriginal groups held identifiable parcels of land by unbroken occupancy from a time beyond which, quite literally, ‘the memory of man runneth not to the contrary’. The titles which they claimed were conceded by all their fellows. There are still some parts of Australia, including some of the regions within which development is planned or actually taking place, in which living Aborigines occupy and use lands that have never been ‘waste and desert’ and to which their titles could be demonstrated, in my opinion beyond cavil, to a court of fact if there were such a court. In such areas if the Crown title were paraded by, and if the Aborigines understood what was happening, every child would say, like the child in the fairy-tale, ‘but the Emperor is naked’.

Since Stanner's lecture about the Great Australian Silence, there has been a noticeable move to include Aboriginal people in histories of Australia, with Henry Reynolds in particular being someone who was inspired by Stanner's lecture to put Aboriginal people closer to the central narrative of Australian history. Mdern history textbooks of Australia such as that by Peel & Twomey show the benefit of that, with the perspectives of indigenous people regularly included.

Anyway, getting to Hancock's book Australia, the copy in my university's library is currently on hold, and I can't find a copy online to verify this, but a couple of different articles or books about Hancock comment on the chapter title 'The Invasion Of Australia' and make it clear that the invasion that Hancock is referring to isn't a military invasion complete with frontier wars, but instead an ecological invasion, figuring out how to harness the environment towards the goal of Western-style agriculture.

In the 1998 Hancock Lecture (an annual lecture delivered in tribute to Hancock, who despite his faults was important in the development of Australian academic history), Ian McLean argued that:

...one of Hancock's major tropes of empire [was] the capture and occupation of the land. His first chapter, 'The Invasion of Australia', did not, as the title might suggest, chronicle the clash of armies, but a battle with the land. The taming of nature by pastoralists became the means of forgetting the history of Aboriginal contact. Here the land was not a resource, but an enemy to be defeated as in any other invasion. Thus, he writes:

The explorers were scouts thrown out by the advancing army of pastoralists . . . Far away on the fringes . . . adventurous pastoralists skirmished with drought and raided the desert . . . The story of these brave assaults upon the interior of Australia . . . that adventurous race of men who first dared, with their flocks and herds, to invade the unknown interior of the continent.

The land and not the Aborigines were invaded and defeated. The Aborigines were not conquered because they had never conquered the land. The Aborigines were not defeated but dispossessed - which is why his opening sentence, 'the British peoples have alone possessed her', immediately writes Aboriginal texts out of the picture without even needing to account for or name them. The Aborigines have no role in the making of Hancock's Australia- that is, they have the role of oblivion.

Elsewhere, in a 2010 biography of Hancock, A Three Cornered Life: The Historian W. K. Hancock by Jim Davidson, Davidson ultimately agrees with McLean:

...after spectacularly entitling the opening chapter of the book 'The Invasion of Australia', it soon becomes plain that hie is tracing the impact of the British on a new environment, which they transform utterly. Although aware of the brutal effect on the Aborigines, Hancock's own attitude seems to merge with that of the average white Australia, who 'sheds over their predestined passing an economical tear'.

So, basically, despite the provocative chapter title, Hancock's Australia indeed fits fairly and squarely within the Great Australian Silence framework described by Stanner - he's talking about ecological invasion. In Hancock's day, it was a common expression that Australia's economy was 'riding on the sheep's back', and Hancock was concerned with showing how the Australian economy was transformed by transforming Australia's ecology into something suitable for merino sheep. So yes, there was no controversy and backlash at the time, because Hancock is not talking about, you know, the topic matter of John Connor's 2002 book The Australian Frontier Wars 1788-1838.

The discussion of Australian historiography was first really started by Bill Stanner in the 1960s. Stanner challenged the narrative of an Australia settled by the British peacefully, addressing what he called the Great Australian Silence. This focus on the negative impacts of colonisation, especially for indigenous Australians, was thread that carried through the works of his contemporaries such as Manning Clark and Henry Reynolds. Clark’s six volume history of Australia, for example, was controversial in its attack on the old colonial values of early Australia and the particular historical narrative it created (I should note that Clark’s views on indigenous Australians were criticised, and he only edited his Histories to include acknowledgements of aboriginal lives quite late in his life).

Australian history was increasingly politicised in the late 80s and early 90s in Australia. While Clark was a huge supporter of Whitlam and virulently anti-Menzies (which is influential in volumes 4 of the Histories). A large part of this was the eminence of Australian historian Geoffrey Blainey, who lectured at the University of Melbourne after Clark. In a 1988 article, Blainey described the current trends in Australian history as negative towards the British, focusing on violence, exploitation, repression, racism, sexism, capitalism, colonialism, and a few other 'isms'.’(Blainey, Eye on Australia: Speeches and Essays of Geoffrey Blainey, 1991). Blainey later fleshed out his views in the 1993 John Latham lecture, coining the phrase ‘black armband history’ for this particular focus (Clark was one of the historians named). The Howard government of 1996-2007 adopted Blainey’s term and weaponised it against critics of the Howard government’s indigenous policy. This included the 1997 Bringing them Home report on the Stolen Generation and a government review of the National Museum of Australia, to name a few issues.The ‘history wars’ as they’ve come to be known, saw an outpouring of historical debate on either side of the political spectrum.

I’d argue that discussions of the invasion of Australia aren’t so much taboo as they are tied to political and social values. Debates around invasion and violence are tied up with ideas of nationalism and patriotism, and ultimately a sense of self. Writing the history of Australia becomes increasingly complex when the very narrative has been tied to a sense of national pride (or a lack thereof).

Sources (some info missing, sorry!)

Blainey, Geoffrey. The Tyranny of Distance. MacMillan, 1968.

Clark, Manning. A History of Australia: I through VI. Melbourne, Melbourne University Press.

Macintyre, Stuart and Clark, Anna. The History Wars. Melbourne, Melbourne University Press, 2003.

Folks who specialize in Australia: OP referenced 1931 here. Did World War II/Adolf Hitler/the Nazis effect public perception of the propriety of committing an "invasion" of Australia? In other words, did World War II create a sense of taboo around being the "invader" in a war, such that Australians who wanted to write a generally patriotic history would have wanted to avoid labeling themselves as "invaders" after World War II but wouldn't have felt that sense of taboo beforehand?

To one very far away, New Zealand appears to have treated the Maoris very differently, and indeed celebrates them.

If I'm not wrong, what accounts for the difference from Australia?