Part of the reason why Botany Bay was chosen as a British colony was the fact that James Cook and Joseph Banks reported that 1770 Australia was very sparesly populated on the coasts and hardly at all in the interior. They believed that Indigenous Australia's nomadism and 'simplicty of life' meant that it would be hardly disruptive or dangerous to establish a British outpost there. They also believed that the 'primitivity' of the Aboriginal people meant that it would be easy to 'convert' them to the settled way of life or defend their settlements - they believed Australia to be an easier target than other available options like outposts in Africa or the Pacific.
When the First Fleet carried 1400+ convicts and solders into the land of Sydney's Eora people in 1788, they already outnumbered the locals, and Sydney was the first of many European invasion 'beachheads' which all played out in similar but unique ways. Direct opposition to invasion was intially rare - Aboriginal Australia often treated the Europeans as lost ancestors or as just another tribe, and tried to adapt to or assimilate these alien invaders. This gave Europeans time to bring in more settlers, claim local resources, establish settlements and reach a local density that the native tribes had no hopes of matching.
For their part, Europeans were under orders to keep the peace, and being seemingly outnumbered by a continent of foes was good incentive. They need not have feared, as hierarchical leadership, armies, invasion or a sense of race or nationhood were not concepts that existed in precolonial Australia. Yet this didn't ease the European fear of attack, which outside of established urban centres like Sydney was constant and justified - on the pastoral or agricultural frontiers, settlers were usually outnumbered and outmatched, and Aboriginal warriors regularly attacked convicts or unarmed groups for reasons of justice which Europeans rarely understood. Conflict was generally centred on the European sense of private ownership versus the Aboriginal understanding of shared ownership - Aboriginals stole European property while Europeans hoarded and fenced off Aboriginal resources. Both groups believed in collective punishment, but Aboriginal warriors rarely attacked more than one man, whereas European reprisals usually targeted many innocents.
By the time conflict did arise, it was usually far too late for Aboriginal Australians to effectively fight back. In the months or years building up to conflict, vital resources like food and water were taken, there was a great deal of death and disorder created by disease and alcohol, and society was culturally disrupted or disunited by things like youths breaking with tradition or war with other tribes. The collapse of Aboriginal Australia was rapid and shocking to sympathetic Europeans, and could even occur before first contact, by tribes accidentally spreading disease and alcohol inland. Frontier warfare, including massacres, policing and exile, usually finished the job, leaving broken refugees providing willing or unwilling labour on the outskirts of white settlement.
As to how Britain managed to attract settlers to Australia, the problem was cost of fare and the perception and consequences of isolation from Europe. Australia was attractive, but North America more so.
People nowadays imagine Australia as a tyrannical penal colony in a harsh climate, but that was not really the case - the punishment was labour and exile to an alien or uncivilised place, but it eventually came to be seen as an opportunity for a new life. Indeed, during the early decades of the 1800s Britons came to complain that the convicts enjoyed their punishment far too much for it to be effective, and that transportation was an expensive gift to ne'er-do-wells. Britain was an industrialising land seemingly overflowing with unemployment and capital - Australia was a land full of resources ripe for exploitation. However, the same could also be said for the USA, which was significantly closer and cheaper than Australia for European migrants, and was still expanding westward. Given the choice, most poor people would pick the US - it was famous, it was more likely to have family, it was 'civilised', and most of all it was affordable.
This isolation from Britain was sometimes a boon. Settlers, ex-convicts and 'native-born' colonials could demand better pay and conditions for their labour - it helped that Britain exiled its Luddites and unionists. Partly founded by political exiles from Scotland and Ireland, it was a more united and accepting society than the one back home. The gentlemen capitalists or administrators were often experienced men of empire or inspired by enlightenment ideals, seeing opportunity in the 'newness' of Australian civilisation. Convicts managed to prosper in an environment where rare skills were in demand, and others married upward and or ran businesses for the gentlemen bourgeoisie. The desire for settlers drove colonial governments to be generous with their land grants - the land around the Swan River was distributed according to the quantity of tools and labourers a wealthy colonist brought with them, and in NSW a squatter could venture out 'beyond the pale' to claim and exploit the land without government grant.
However, it also created difficulties besides isolation or high labour costs. The journey to Australia was long and hard and incredibly uncomfortable, and was so expensive that it was usually only single men making the journey - Australia was seen as a dangerous frontier 'mans land', both at home and abroard, and its ratio of men to women did not equalise until the First World War. The cost also meant only the richest settlers returned to Britain, and that importing and exporting was so expensive that it stifled industrial growth and made the colony incredibly self-reliant. News from Britain was generally many months out-of-date by the time it arrived in Australia, and the colonies were indefensible should the US, France or Russia attack.
Eventually, a solution to funding cheaper migration was dreamt up by a British man named Edward Gibbon Wakefield - he suggested that land be sold at high prices by colonial governments, who could then use this income to subsidise lower class migration. Wakefield himself was highly influential in the founding of South Australia and in New Zealand, and although protested by those upper and lower class Australians who benefited from the previous system of high wages or cheap land grants, every colony adopted his policy and it worked a treat. Whaling and sealing were Australia's first industries prior to Wakefield's reform, but struggled to find workers - they were quickly outshone by wool and gold, which alongside cheaper fares and increasing capital from London drove enormous growth in Australia's colonial cities and expansion into its frontier.
The best example of this growth is Melbourne, which was found as a backwater sheep town by Tasmanian pastoralists in 1835 - thanks to several goldrushes slightly inland, within 2-3 decades it was one of the world's wealthiest, glamorous and fastest growing cities. The goldrushes across southern Australia transformed its urban centers from colonial outposts into prosperous Western cities that were highly desirable places to live, favourable compared to the crowded poverty of New York. At the same time, improvements in sailing and shipbuilding, and reforms to laws concerning long distance travel by sea, made the journey to Australia safer, faster and far more comfortable. Although still more expensive, it was generally considered more comfortable than the cramped and borderline abusive conditions of the American migrant ships.
Sources:
Tyranny of Distance by Blainey
The Other Side of the Frontier by Reynolds