Spain never had a colonial presence in New York City. The Dutch did until 1664 (and again briefly in the 1670s) and then it remained an English colony until American independence in 1776. Actually, until 1783, since New York City was under British occupation for the duration of the war. The war ended on Evacuation Day, November 25, 1783, the day the British evacuated New York City per the terms of the peace treaty, and George Washington and the Continental Army triumphantly, and symbolically, marched into Manhattan.

There are quite a few buildings left in New York and New Jersey from the colonial period. For as nearly a comprehensive listing and description of them as there is, you may want to track down the out-of-print books Pre-Revolutionary Dutch Houses and Families in Northern New Jersey and Southern New York by Rosalie Fellows Bailey, and Dutch Houses in the Hudson Valley Before 1776 by Helen Wilkinson Reynolds.

However, few colonial-era buildings exist in Manhattan, where the original New York City existed. The oldest building in Manhattan is the Morris-Jumel Mansion, in Washington Heights, built in 1765. But this building was far outside the city limits when it was built. At the end of the war, the actual city part of New York City was the area between Water Street and the Bowery, and mostly below Wall Street. There was also Greenwich Village by then, consisting of a couple dozen buildings, as well as Harlem, which was a town center of maybe a dozen more. Everywhere else on Manhattan was farmland, dotted with isolated farmhouses. The only part of the city that could be preserved to provide its colonial character is the area below City Hall, or, really, Wall Street.

So what happened? A few events made it unlikely for much of anything to survive:

• In 1776, the British took occupation of the city. Patriots still residing in the city are thought to have started a fire to burn it down, to make the city less useful to the British forces. Regardless of who started the fire, the fire is estimated to have destroyed 10-25% of the buildings then existing within the city. Of the rest, the British took possession of many surviving buildings for military purposes, and did not maintain them, often damaging them structurally to suit the purposes of what the military decided to use them for during the war. This led to a lot of them being unusable for their original purpose after the war. When the owners returned to the city after the war, they chose to rebuild rather than restore. Other owners never returned to the city at all, having built new homes upstate, or even as far away as Philadelphia. They simply sold their land in Manhattan to the highest bidder, with the original buildings being demolished.

• In 1811, the city approved the Commissioners' Plan that laid out the grid system for the city, encompassing most everything above what is now Houston Street (with Greenwich Village being the main exception, though there are a couple of isolated streets that survived, notably Broadway). The grid was laid out over the course of many decades, and wasn't complete until the late 1800s. Throughout the period, the large landowners sold off their land to developers, who generally demolished the buildings so that the new buildings would align with the street grid. Other buildings had to be demolished because they stood directly in the path of one of the planned streets on the grid.

• In 1835, there was a second Great Fire of New York City, estimated to have destroyed 500-700 buildings. Once again, the fire occurred downtown, started at a warehouse located at the intersection of Wall Street, Beaver Street, and Hanover Square. Just about every building in a seventeen block vicinity had to be demolished.

• In 1845, a third Great Fire of New York City broke out, destroying an estimated 350 buildings. This one started a few doors down from where the New York Stock Exchange building now is. It spread north of Wall Street and all the way down to Bowling Green.

• One further note: the area below Water Street didn't exist before 1776. That area was built on landfill in the 1800s.

Between the 1835 and 1845 fires, then, virtually the entire city that existed before 1776, east of Broadway, was burnt down. And the area west of Broadway is smaller, and was where the original military fort was, as well as the battery, which took up a lot of the area. There is very little that could have survived, and none of it really did.

The only Dutch colonial building thought to have survived all three fires is the Rigging House. The year after the third fire, in 1846, the lithographer H.R. Robinson created a lithograph of the building, which was at 120 William Street, between Fulton and John Streets. The building had been used for a while starting in 1748 as a Methodist church, before the Methodists built a proper church. By 1846, it was the home of an engraving company, and in poor shape. It was demolished in 1850.

Only one confirmed building that was actually within the city limits of urban New York City survives from the pre-independence period. That is St. Paul's Episcopal Church on Broadway, which has been dated to both 1764 and 1766. In addition, there is the Captain Rose House, at 273 Water Street, which may have been built as early as 1773, according to the New York City Landmark Preservation Commission. However, it was probably built either around 1781, or 1793, and was certainly built no later than then.

There are also a small handful of buildings that survive from the early "Federal Period", between 1783 and the first fire in 1835. Below Wall Street, the oldest building is Castle Clinton, inside Battery Park, which was built in 1811-12, and was the original immigration station before Ellis Island. Also downtown, but north of Wall Street, are the James Brown House in Greenwich Village, built before 1817, probably in the late 1700s, after the war and British evacuation; the Bialystoker Synagogue on the Lower East Side, built in 1826; and the Old Merchant's House just off the Bowery, built in 1832.

Notably, of these, only St. Paul's Church and the Captain Rose House were anywhere near the path of any of the three fires.

There are also a couple of recreations. Fraunces Tavern dates itself to 1719, though it's actually a restoration done in the first decade of the 1900s, based upon its known structure in the early 1800s. Its original structure from the 1700s was unknown at that time, and is still unknown. Another restoration is found not far away at 57 Stone Street, a 1903 recreation of the Dutch colonial style (notice the Dutch colonial roof). There is a similar building on the same block. The whole block is restored to look like what Manhattan would have looked like before the Revolutionary War, though none of it is actually authentically from the 18th century and before.

Uptown, above 14th Street, there are a few authentically surviving buildings from the 1700s and early 1800s. The aforementioned Morris-Jumel Mansion (1765) in Washington Heights is the oldest. There is also the Dyckman House (1785) in Inwood, the only surviving farmhouse in Manhattan, built in the Dutch Colonial style, an artifact of its builder being a Dutch New Yorker; Gracie Mansion (1799) in Yorkville, which is the official residence of the Mayor of New York; and the Hamilton Grange Mansion (1802) in Harlem, the home of Alexander Hamilton.

Outside of Manhattan, there are quite a number of buildings dating to the 17th and 18th centuries. But because the outer boroughs were all farmland until the 19th century, these are all isolated buildings, and surrounded by modern architecture, so there isn't any large pocket to look at to point to as "colonial era New York". Among them are the Pieter Claesen Wyckoff House (1652), in Canarsie, Brooklyn, the oldest building in any of the five boroughs; the John Bowne House (1661) in Flushing, Queens; the Billiou–Stillwell–Perine House (middle section from 1662) on Staten Island; the Britton Cottage and the Abraham Manee House (both 1670) also on Staten Island; the Schenck House (c.1675), dismantled and rebuilt inside the Brooklyn Museum; the Conference House (c.1675) on Staten Island; and the Old Quaker Meeting House (1694) in Flushing, Queens. There is also the Old Stone House in Park Slope, Brooklyn, which is a replica of a 1699 building important to the Revolutionary War, rebuilt in 1935.

For further reading on the fires, British occupation, and the Commissioners' Plan, a one-stop reference is the book Gotham: A History of New York City to 1898 (Oxford University Press) by historians Mike Wallace and Edwin G. Burrows.

TL;DR: British occupation and a major fire in 1776, followed by two more major fires in 1835 and 1845, burned down pretty much anything from the colonial period that existed within the true city limits of New York City as it existed before independence. The construction of the grid layout demolished more. Some colonial buildings do exist uptown and in the outlying boroughs, all of which was farmland/small farm towns until the 1800s, so they exist as isolated buildings surrounded by more modern architecture.