The settlement site was chosen because freshwater was available. The water supply was a creek now called Tank Stream (and today it's an underground stormwater drain). The catchment area for Tank Stream covered much of the current Sydney CBD. The name comes from reservoirs ("tanks") that were cut into the creek in 1790 to provide water storage.

Tank Stream was the main water supply for Sydney until, running through the middle of the settlement, it became the main sewer for Sydney. This new purpose reduced its quality as a water supply, and water had to come from wells or be brought in from further away, until a new water supply, Busby's Bore, started operation in 1830.

Early settlements in Australia were usually chosen for their combination of a good anchorage for shipping and a reliable freshwater supply. In some cases, the initial site was abandoned because the water supply proved unreliable. This occurred in Tasmania, where the first site for settlement in 1803 was Risdon Cove, on the east side of the Derwent estuary. Early in 1804, the settlement moved to Sullivans Cove, on the west side of the estuary. Similarly, in Brisbane, the first settlement was at Redcliffe in 1824, but moved to North Quay in the Brisbane River estuary in 1825 (which is the current CBD).

Other requirements for successful settlement included land suitable for farming, good drainage and security against flooding, and access to the surrounding area by land. Of all of these requirements, only good anchorage could be guaranteed - suitability for farming wasn't certain until the first crops were grown, drainage and flood risk might only be apparent in the first heavy rains, and a water supply initially thought suitable might dry up in the dry season.

References:

NSW Office of Environment & Heritage, "Tank Stream": https://www.environment.nsw.gov.au/heritageapp/ViewHeritageItemDetails.aspx?ID=5045604