As you probably know, New York was founded by the Dutch. And the original Dutch colony of "New Netherland" (New Amsterdam was just the town at the tip of Manhattan) included New York State, New Jersey, and stretched all the way down to Delaware and the Philadelphia area.

The eastern half of Long Island (today's Suffolk County, New York) was originally part of the Massachusetts colony, while Connecticut was its own colony founded in 1636 under the English.

After the English conquered New Netherland in 1664, the king of England Charles II gave the land to his brother, the Duke of York (who would later become the king of England himself as James II). It was renamed New York.

This was just a few years after the English Civil War and the restoration of the English crown. As repayment for loyalty to the crown during the war, Charles II granted the part of New Netherland west of the Hudson River to a couple of lords named Berkeley and Carteret. The original land grant stated that the northern border of this new province, now renamed New Jersey, was the northmernmost point of the Delaware River system. The Delaware flows as far north as today's Point Jarvis, New York, which is today a border town between NY and NJ.

New York City stayed with the province of New York and under control of the king's brother because that land was more valuable. The Hudson River was the key waterway of early American settlement. Via the Hudson, you could get to the Mohawk River which could carry you all the way to Lake Ontario and the St. Lawrence River where the valuable fur trade was based.

So the king's brother got all the best land of former New Netherland, the king's friends got all the (mostly) unsettled land of the former Dutch colony, while the English colony of Connecticut remained its own thing.

There were several skirmishes over exactly where the south New York/north New Jersey border was for several decades before the land was more thoroughly surveyed and the border was more properly defined.