First off, a municipality's boundaries are fairly arbitrary and tend to sprawl out until another city that was previously distant to the city "collides" with it.

Secondly, New York's is composed of an extremely dense vintage building stock that predates and would be illegal under most cities' modern building and zoning codes.

But the big reason is annexations, which greatly succeeded in New York.

http://en.wikipedia.org/wiki/City_of_Greater_New_York

The City of Greater New York was the unofficial term for the expanded City of New York created on January 1, 1898 by consolidating the existing City of New York with the East Bronx, Brooklyn, most of Queens County, and Staten Island. The section of the Bronx west of the Bronx River had been annexed to the City and County of New York in 1874, and was known as the Annexed District. In the years leading up to consolidation, City of Brooklyn had expanded by annexing all of the other towns and cities in Kings County. Only the western part of Queens County was part of the consolidation plan.

Chicago annexed as well, but hit a point where extending city services was no longer practical. Los Angeles annnexed plenty, but the towns it had gobbled up were far more suburban in nature compared to the density in New York.

Ultimately, density is the key factor--after the advent of the automobile, Americans started to turn away from packing in near streetcar and other mass transportation lines and suburbanization and space became preferential. The cities that New York annexed were largely constructed well before the automobile.