/u/uncovered-history has provided a previous answer about why Jamestown didn't work out. The town was built on a swamp, which made the nearby seaport town of Norfolk and the adjacent community of the "Middle Plantation" (later renamed Williamsburg) to be more attractive settlements for newcomers. It was also mostly burned to the ground during Bacon's Rebellion and then the statehouse burned down again in 1698. Virginia then decided to take the opportunity to move the statehouse to the higher ground in Williamsburg, more centrally located between the James River and York River.

Plymouth and Salem didn't grow like New York because Boston did instead. Plymouth was founded in 1620, and while it has a satisfactory seaport, it isn't built on any river system with access to the interior of the colony/state. Salem was founded in 1626, and while it is adjacent to the Crane River, that river doesn't provide much access to the interior, either.

Boston, then, became the major seaport in Massachusetts (which was, at first, the two separate colonies of Plymouth and Massachusetts Bay, but founded by very like-minded Puritan populations; the two were merged into one in 1691 after the Boston Revolt). The oldest part of Boston is Charlestown, founded in 1628, while the original town of Boston itself was founded two years later, in 1630. Before 1630, the population of the whole Plymouth colony was only about 350-400 people.^1 The population in the Massachusetts Bay colony was even smaller. The Winthrop Fleet arrived in 1630, carrying upward of 1,000 new colonists with them,^2,3 which alone outnumbered the pre-existing English population in the two colonies combined. The next decade marked the height of the Great Migration of the Puritans to New England, when an estimated 20,000 new colonists arrived to the area,^4 1,200 of whom settled in Boston.^5 The initial populations of Plymouth and Salem weren't much more than a blip in comparison to this influx that began a decade or less later.

One of these arrivals was William Wood, who published a portrait of Massachusetts and Plymouth in 1634 called New-England's Prospect. The aim of the book was to encourage new, potential English colonists to immigrate. In the book, Wood describes Boston Harbor as the most natural harbor in either colony, with Boston being the area's preferred seaport already (spelling modernization mine):

"This Harbor is made by a great company of islands, whose high cliffs shoulder out the boisterous seas, yet may easily deceive any unskillful pilot; presenting many fair openings and broad sounds, which afford too shallow waters for any ships, though navigable for boats and small pinnaces. The entrance to the great Haven [i.e., Boston Harbor] is called Nantascot [now Hull, Massachusetts]; which is two leagues from Boston; this place of it self is a very good Haven, where ships commonly cast anchor, until wind and tide serve them for other places; from hence they may sail to the River of Wessaguscus [i.e., Weymouth Fore River], Neponset, Charles River, and Mystic River, on which rivers be seated many towns."^6

It's a bit unclear if Wood is referring to Hull or Boston as the "very good Haven" for ships to anchor, but it's most probably Hull. Nevetheless, that didn't last long, and Boston became the preferred seaport. Wood mentions the "many towns" along the rivers flowing into Boston Harbor, and it's the Charles River which provided many of the major inland settlements of the Massachusetts and Plymouth colonies in its early years. The Charles River is the most navigable river, and easily gives more access to the Massachusetts interior, than any other river that flows out into Boston Harbor. It also provided access to a freshwater source to more colonists than any other river could. Not only that, but Boston was built between both the Charles and the Mystic rivers, so colonists along the Mystic relied on the Boston seaport, too. Thus, Boston could service more farms and communities than any other seaport on Boston Harbor, or on the Massachusetts and Plymouth seaboards in general. Boston quickly eclipsed any of the other seaports of either colony.^7,8,9

That said, that's not to say Plymouth and Salem didn't grow at all. They were both among the 25 largest U.S. cities on the 1790 Census, and while Plymouth fell off rapidly after that, Salem didn't fall out of the top 50 cities until after the Civil War. They both continued to grow, Salem in particular, just nowhere near as fast as Boston did, as it became the main seaport and commercial trade center. And it quickly fell behind many other major settlements as the U.S. expanded westward.

As for why New York grew and not any other city in New York, the answer is pretty much the same as it is for Boston: geography. Manhattan lies in the middle of New York Harbor, at the mouth of the Hudson River. The Hudson River system provides better access to the interior of the North American continent than any other on the Eastern seaboard. In fact, New York City is the only point in New York that provides access to the interior of the colony/state from the Atlantic Ocean, and New Jersey doesn't have anything comparable to the Hudson. The rare hurricane aside, Manhattan is not very prone to flooding, and New York Harbor rarely freezes over, making it one of the best natural harbors in the world, certainly in North America. That is only one reason why New York ended up dwarfing any other city in the U.S. population-wise; I have written a more comprehensive answer on this subject in this sub before. But when limiting the comparison to New York City vs. Jamestown, Plymouth, and Salem, the difference is New York has the geographic advantages that the others do not, which resulted in those other settlements being abandoned/overtaken in favor of more preferable seaports nearby.

^1 Stratton, Eugene Aubrey. Plymouth Colony, Its History & People, 1620-1691 by Eugene Aubrey Stratton, 1986, p.50.

^2 Winthrop, John, and James Savage. The History of New England from 1630 to 1649, 1853, p.442.

^3 Morgan, Edmund S. The Puritan Dilemma: The Story of John Winthrop, 1958, p.55. While Massachusetts Gov. John Winthrop himself estimated the number arriving with his fleet at 700, Morgan estimates that around 1,000 immigrants arrives in 1629-30. But Morgan also notes that upward of 200 of these new arrivals died during their first winter there, and as many as another 200 returned to England the following spring.

^4 Baines, Dudley. Migration in a Mature Economy: Emigration and Internal Migration in England and Wales 1861-1900, 1985, p.57.

^5 Friedman, Karen J. "Victualling Colonial Boston", Agricultural History, Vol. 47, No. 3 (Jul 1973), p.189.

^6 Wood, William. New-England Prospect, 1634, p.3.

^7 Weston, George F. Boston Ways: High, By & Folk, 1957, p.10-15.

^8 U.S. Government Printing Office, pub. Boston Harbor Massachusetts Deep Draft Navigation Improvement Project, Part 3 of 3, 2014, p.595-597.

^9 Rawson, Michael. Eden on the Charles: The Making of Boston, 2010, p.8-21.