The majority spoke Dutch and it was the franca lingua but it was never a universal language. New York City has been a very multicultural place since the very beginning. Peter Minuit, who famously bought Manhattan from the Indians on behalf of the Dutch West India Company, was himself a Walloon (the French speakers in what is now Belgium).

And the first ship of settlers that came over after that were also Walloons--they had lately been refugees in the Netherlands at a time of great religious persecution against non-Catholics in France and Spain.

Some of the next ships over contained shipbuilders from Scandinavia. Shipbuilding was a very necessary skill to have in the New World and the Dutch West India Company paid many shipbuilders to make the move.

Therafter, most newcomers were from the Netherlands. However, by the time the English first seized the colony in 1664, there were many other languages being spoken. In addition to the aforementioned Dutch, French, Norwegian, and Danish immigrants, there were also speakers of German, English (the town of Gravesend, which became part of Brooklyn, was founded by English Quakers in 1643), Arabic (or something like it from North Africa, since there were some Muslims living in the colony by then), Sephardic Hebrew (a shipload of Sephardic Jews settled in 1655), and several others. The slave trade had come to New Amsterdam by 1664, so several African languages were spoken by then.

The English government immediately made English the official language of all government communications, including in the courtroom. However, Dutch continued to be the dominant language well into the 1700s. By the mid-1700s, it still made up a plurality though by then many of the American-born Dutch speakers were bilingual. By the time of the American Revolution, English was the language a majority of New Yorkers spoke, and after the war, the Dutch population began to disband. The Dutch towns in eastern Queens, up the Hudson River Valley, and in northern New Jersey had become home to English-speaking refugees during the war because the British military had seized and occupied the city. Many of those English speakers remained living in the Dutch countryside after the war, and then the new U.S. government opened up the Western frontier for settlement, and many of the younger Dutch--by then, often intermarrying with their English-speaking neighbors--left New York for better opportunities. By the mid-1800s, there weren't really any "Dutch towns" left, though the last American-born Dutch speakers in rural New Jersey were still around in the first few decades of the 1900s.