If I understand correctly, what made New York City -- all five boroughs -- as wealthy as they are today is the natural harbor.
What I'm struggling to understand is why Manhattan was where people were drawn to. If the harbor is indeed the sell, why not Brooklyn or Staten Island, something closer to the ocean? Why sail a ship all the way into the harbor to Manhattan when you could stop in Bay Ridge, Staten Island, or even Long Island more quickly?
This is explained in Russell Shorto's book The Island at the Center of the World.
The Dutch set up three forts in 1624-25:
One of them was called Fort Wilhelmus. It was built on the Delaware River near present-day Burlington, New Jersey. It was intended to be the main settlement, under the mistaken belief that the climate there was warmer than it actually is. The first winter, the river froze. The next winter, it froze again. They abandoned it before the third winter.
A second one was called Fort Orange. It was located near present-day Albany, New York, near where the Hudson meets the Mohawk river. The Dutch had established a fort here ten years earlier, but abandoned it due to flooding. This time, it was abandoned after the Dutch found themselves on the wrong end as allies in a war between two local Indian nations.
The third one was called Fort Amsterdam. It was initially built on Governors Island (or "Nut Island" as they called it) in New York Harbor, the small island south of Manhattan. This small island was the most easily defensible position while the Dutch were living as squatters with no land claim.
The immediate problems with Governors Island as a long-term residence were that it had no pastures where the livestock could graze, and it had no fresh water. While Brooklyn is geographically the closest borough to Governors Island, the part of Brooklyn that is adjacent to the island was very hilly. It's the part of Brooklyn that gave the neighborhoods of "Cobble Hill" and "Brooklyn Heights" their names, and nobody wants to walk their livestock up and down a hill every day. There also wasn't a good freshwater source right there. The closest freshwater sources in Brooklyn were Gowanus Creek, or Bushwick Creek.
By ferryboat, both were further away than Collect Pond in Manhattan, in present-day Chinatown. There were also multiple streams that fed into the pond, and an adjacent pasture. As such, almost immediately upon arriving to colonize the mouth of the Hudson, the Dutch colonists found themselves ferrying back and forth to Manhattan for daily necessities.
Within the first two years of these three Dutch colonies, all three were on the verge of collapse. This mismanagement led to a revolt by the colonists against the colony's director Willem Verhulst. Verhulst was replaced by Peter Minuit, whose name you may have heard of. Minuit arranged a sale for the land at the south end of Manhattan, and, with that accomplished, he ordered Fort Orange and Fort Wilhelmus to be abandoned. All the colonists were to convene as the pioneer settlers of a new colony on Manhattan island.
Shorto explains that Verhulst's main job for the Dutch West India Company while he was there was to survey the geography in rather excruciating detail, and make recommendations to the company on how to proceed with future settlements. When Verhulst did nothing about making any purchases, he was removed and Minuit began negotiations for Manhattan almost immediately. Though there is scant evidence to say why Manhattan was chosen, Shorto uses what's available to conclude it was most likely chosen based on the immediate geographical necessities of the colonists:
To anyone with a practical and logistical mind it was clear that the island of Manhattan, separated from Nut [Governor's] Island by a channel “a gunshot wide,” answered every need. It was large enough to support a population, small enough that a fort located on its southernmost tip could be defended. Its forests were rich in game; it had flatlands that could be farmed and freshwater streams. It was situated at the mouth of the river to which Indian fur-traders came from hundreds of miles around, and which connected to other waterways that penetrated deep into the interior. It was also at the entrance to the bay, located in a wide and inviting harbor that seemed not to freeze over in winter.
However, this was only the first land purchase by the Dutch, not the last. In fact, the next land purchase was also by Minuit, and it was of a piece of Staten Island. More purchases and settlements followed in the 1630s and early 1640s, so that there were Dutch settlements on all four sides of Manhattan. It was still possible one of these other settlements could have "won out" and become the main European settlement in the area.
But beginning in 1643, the Dutch engaged in a series of wars against a succession of local Indian nations, more than one of which the Dutch instigated. These include Kieft's War in 1643, the Peach Tree War in 1655, and the two Esopus Wars in 1659 and 1663. Kieft's War was particularly devastating, virtually wiping out all the settlements surrounding Manhattan up until that time, the colonists fleeing to Fort Amsterdam in Manhattan for safety. Not all of the destroyed settlements were rebuilt, and of those that were, several more were abandoned again due to destruction that occurred during the Peach Tree War.
It was really only after the English took over in 1664 that the Europeans were secure enough to defend any of these outlying settlements, and even that was still in flux until 1675, when the dispute over the colony between the Dutch and the English was resolved once and for all. By the time the Europeans really had the population and military investment to establish anything permanent outside of Manhattan in the harbor, Manhattan had a fifty year head start building infrastructure and community.
As of 1898, the outer boroughs were still playing catch-up, and the consolidation of Greater New York City that year essentially ended any sort of competition, uniting the boroughs as one.
The impetus for Manhattan's initial Dutch settlement was to protect the mouth of the Hudson River and the fur trade upriver. As OP stated, the natural harbor provided an optimal place for a fortification. The Dutch in fact settled many locations along the harbor and across the region, but ultimately chose Manhattan for their main settlement.
The Dutch first settled the Hudson River region following Henry Hudson's voyage up the river in 1609. Foremost was the fort in present day Albany (constructed 1614), but colonists also settled in lower Manhattan, Harlem, Governor's Island, Long Island, and along the Hudson River valley. Albany is located on the banks of the Hudson at the northernmost navigable point for oceangoing vessels, much farther inland than a ship could travel on any other river on the east coast. This was an ideal place to service the fur trade, the initial aim of the Dutch colony. During that time the Dutch also constructed a fort along the Delaware River to the south and started settlements in present day Connecticut to the north and east.
Attention turned to Manhattan after 1624 and the arrival of a ship carrying thirty Walloon families, French-speaking refugees from modern Belgium. The majority of the group settled in Albany, but a smaller number were initially dropped off at Governor's Island (then Nut Island). Within the next year, however, the Dutch West India company would send a contingent lead by Willem Verhulst with instructions to move the Governors Island settlers to the southern tip of Manhattan and construct a fort there. The fort, called New Amsterdam, was to be the administrative center of the company's New Netherlands colony, of which Verhulst was the director.
To answer OP's specific question, the area's geography can explain the choice of Manhattan. The outer harbor, south of the "narrows," can be ruled out because it faces the open ocean. It's hard to say exactly why settlers didn't choose each possible location in the inner harbor, but there are steep cliffs along much of the New Jersey side of the Hudson as well as a steep embankment on the Brooklyn side of the harbor (Brooklyn Heights). Some of persent-day Brooklyn was settled around that time, but it was a scattering villages that were part of the colony's agricultural hinterland. Staten Island is located farther out in the harbor and, partially due to conflicts with the Indians who lived there, the Dutch weren't able to form a permanent settlement in the area until decades later.
Centrally located and much larger than Governor's Island, Manhattan offered space for multiple settlements and farms and provided land access to all the region's waterways. A fort at its southern tip can equally protect the mouth of the Hudson and East rivers.
All of this was of course well known to the region's residents for centuries before the Dutch arrived. During their voyage, Hudson's party recorded many encounters with Indians who lived across the region's islands and navigated the rivers and bays in canoes while trading with the Dutch. Notably, at the time Manhattan's southern tip was used as a neutral point where different tribes met for trading.
So it stands to reason that under the command of Verhulst and later Peter Minuit, the Dutch consolidated many of their disparate settlers into their new Manhattan location, ultimately vacating almost all settlers from the colony's other forts. Historian Oliver Rink, writing about this choice says,
Minuit may have decided to concentrate settlement on Manhattan because the settlements on the Delaware and Connecticut were indefensible against Indian attack, and wind conditions on Long Island Sound and along the Jersey coast often delayed the coastal sloops for days as they tried to resupply the outposts of European settlement.
Edit: I started writing this before /u/lord_mayor_of_reddit's response, and I tried to edit it down so there's less overlap.
Sources: