New York City was originally the town at the southern tip of Manhattan. During the Dutch period, it grew to about where Wall Street is, and during the English/British period, it grew to about where City Hall now is.

But also during the colonial period, other towns sprouted up in other parts of Manhattan. Greenwich Village, or simply Greenwich, was one of them, just north of New York City, on the west side of the island. Further uptown was Harlem, or Harlem Heights, which was a small village of a couple dozen buildings including a church. There were a few other little village squares throughout the island, usually convenient to several area farmers, who would build their houses near each other because isolated farmhouses were susceptible to Indian attacks.

But most of everything north and east of City Hall and Greenwich Village were farms. And the roads they used were mostly old Indian trails. You can see the old Indian trails and original colonial roads in black in this low resolution map from 1807 here.

In 1776, Manhattan and most of Long Island were captured by the British during the American Revolution. The British made New York City their base of operations for the war effort. But during the battles that led to the occupation--the Battle of Long Island and the Battle of Harlem Heights--about half of all the buildings in New York City burned down, and Harlem was virtually destroyed. There weren't any major settlements north of Greenwich Village anymore.

In the decades following the war, city leaders began considering how the city of New York should develop northward, with immigrants already pouring in to the small city at the south end of Manhattan.

This led to the Commissioner's Plan of 1811, which laid out the grid system of Manhattan with its wide avenues and predictable streets. It was built over a several decades, with surveying completed by 1817, but the plan not fully implemented in the northern reaches of Manhattan until after 1870. For instance, you can see in illustrations of how the corner of 42nd Street and 2nd Avenue looked in the 1860s (1, 2) that the grid may have been laid out, but there was still more development to be done.

The Commissioner's Plan didn't tear out all the old streets. "Old" New York, below Canal Street, was left intact just the way it was, as was Greenwich Village. The two were essentially contiguous by the time the plan began to be put into action. For the rest of Manhattan, the plan retained some of the older streets where it could. Notably, Broadway is not part of the grid plan for much of its length. The section of Broadway between City Hall and 14th Street was new (the original road veered east), but the rest of Broadway essentially follows the same path as the Wickquasgeck trail, which was already there before European colonists arrived. Though, for stretches of it uptown, it was straightened out substantially.

The Bowery was a road built in the 1600s that had originally led to the bowery (farmhouse) of the last Dutch director of New Amsterdam, Peter Stuyvesant, and that road continues on to where Union Square now is, with part of it designated 4th Avenue. And coming off the Bowery are Astor Place and Stuyvesant Street, originally one road built before the grid that intersected the Bowery road.

North of Central Park, there remain many other roads also based on the original Indian paths and organic roads built before 1811. St. Nicholas Avenue, St. Nicholas Terrace, Convent Avenue, Fort Washington Avenue, and parts of Riverside Drive all predate the grid.

But between 14th Street and 125th Street, virtually every pre-existing road was wiped out, aside from Broadway. One reason they could do this was that two of the other main roads in this area could be readily incorporated into the layouts of 8th Avenue and 23rd Street.

Another reason they could do this was that much of this area was made up of large estates, farmland owned by the wealthy elite of New York City. Very little of it had been developed beyond large plots of farmland, each occupied by a handful of tenants, who made use of only a few major roads, particularly Broadway. For instance, the present day neighborhood of Chelsea was, in 1811, a single estate called "Chelsea", situated along Broadway and owned by Clement Clark Moore (the same guy who wrote Twas the Night Before Christmas). Moore and his ilk were people who, for the most part, could afford to take a short-term financial loss for a long-term gain.

The plan wasn't implemented all at once. Between 1812 and 1817, the city very diligently surveyed the entire island, led by surveyor John Randel Jr., who mapped out exactly which pre-existing pieces of land would need to be acquired from individual landholders. His maps showed what plots of land would need to be split up and, in some instances, where entire farmhouses would need to be torn down.

The city then began implementing the plan from south to north, acquiring the land they needed piece by piece. There was certainly pushback from many of these wealthy estate owners, and from more modest landowners, too. But as the plan was implemented, the estate owners and small-time farmers soon found their land to be much more valuable than it ever had been as farmland. Real estate investors were trying to buy whole square blocks to develop into housing for the citizens and new immigrants residing in the congested city downtown. Business owners were interested in developing land for industrial or commercial use. As the grid approached each estate, those pieces of land tended to become the subject of land speculation, ultimately earning owners and investors substantial amounts of money over time.

That's how the city was able to make the grid happen: it was lucrative for the land owners. Some may have experienced short term losses, because nobody wanted to rent farmland that was going to be bulldozed in the coming years, or because a tenant might be displaced when a road would run through the farm and they would be forced to move elsewhere, resulting in loss of income for the landlord. But once the land had been integrated into the grid, the value increased, as investors became interested in developing it for urban use.

As the population of the city swelled in the first half of the 1800s, and the population density in the developed part of Manhattan increased tremendously, there came calls to portion off part of the city as a park, to alleviate the urban immersion that was planned for the island. These calls started in 1844 with the publication of an editorial by William Cullen Bryant, editor of the New York Evening Post. Bryant would periodically continue this advocacy in his newspaper, joined by Horticulturalist magazine, which published "A Talk About Public Parks and Gardens" in 1848 and "The New York Park" in 1851. Thereafter, other New York papers began joining these calls, such as the New York Times. This resulted in the development of Central Park, beginning in 1857, displacing a small village called Seneca. Central Park became the most notable divergence of New York's urban development from the original Commissioner's Plan of 1811. Most of the rest of Manhattan was developed according to plan.

I have written a couple of other posts in this sub related to this question here and here, both with further sources. You can read the original reasoning of the New York City Commissioners for laying out the grid here. These commissioners are who "the Commissioner's Plan of 1811" was named after. Also of interest may be The Geological History of Manhattan from 1843, which describes a lot of the land formations (hills, valleys, ponds, etc.) of Manhattan prior to the implentation of the Commissioner's Plan. The March 1867 edition of Historical Magazine published testimony from an 1857 court case of John Randel Jr. here. Randel was the surveyor for the Commissioner's Plan, and produced the maps used by the engineers and construction crews to lay out the grid. His testimony describes how he went about making those maps and how the roads would then be constructed.

Perhaps most interestingly, in 1864, Randel wrote an article entitled "City of New York, north of Canal street, in 1808 to 1821" in which he describes many of the major roads and land formations that were altered or eliminated with the implementation of the Commissioner's Plan under his direction.

For further reading, try The Greatest Grid: The Master Plan of Manhattan, 1811-2011 by Hilary Ballon, or City on a Grid: How New York Became New York by Gerard Koeppel.

I am curious if the origins of NYC's grid system are the same as Philadelphia's (at least the original boundaries of the city; it gets wonky beyond Vine/South and Front/30th). My understanding is that NYC was settled by the Dutch whereas Pennsylvania principally by Germans (beware the misleading term Pennsylvania Dutch).