Looking at pictures of Manhattan from a century ago, the island was filled with ships from what seems like Central Park to the southern edge. Films set in WW2 also show this. Why and when did this change?
The book below is a fantastic read that directly addresses your question...
The shortest version is ports (anywhere) are a function of the economy, the local labor market, and the entire supply chain of delivering "stuff" from A-to-B.
Once containerization took over post-war, the economics quickly and irreversibly shifted in favor of inland factories.
So no more factories in New York City meant there was no need to maintain and service the piers to bring raw materials to those factories. The factories were only built adjacent to the piers because it was so time consuming and expensive to transfer raw materials. (Even the layout of Manhattan's "grid" of streets took this into account, with the avenues more tightly spaced on the coasts than in the middle of the island. This allowed for good to move more efficiently between factories as they came off the ships.)
Now, what do I mean by "take over"? I'd have to read the book again, but I recall containerization allowed for moving most goods at, without hyperbole, ~10% the cost of traditional break bulk shipping. If it could fit inside a container, you could move it for really, really cheap. No one involved in shipping could ignore the economic advantages of containerization. And those who tried to resist slowly bled out as "the box" took over.
(Break bulk is where individual pieces are carried on/off a ship. Think "On The Waterfront" and a bunch of people carrying bags on their shoulders. With containers, what previous could take a week could be done in less than a day.)
You'll note the shipping actually increased too...but it shifted to New Jersey, which constructed new piers better suited to containerization rather than break bulk shipping. By the time Manhattan and Brooklyn tried to make new piers, it was too...the containerization infrastructure was in place on the mainland.
But what about the longshore men? Did they have jobs to do after the piers went away? Well...automation is always rough on manual labor. Lots of longshore men lost their jobs. Crane operators were the new king. The book is an interesting history lesson in the history of American labor/management relationships, particularly in how containerization was handled differently in New York vs. San Francisco, and even Manhattan/Brooklyn vs. Newark.
Here's a link to the book if anyone is interested. It's well written for even a novice and consumable for a non-academic. The writing style is not overly dry either. There is a narrative to understand. The bibliography is huge too. It's basically the history of shipping "stuff", which in turn answers the question "When and why did Manhattan's ports go away?"
EDIT: I'll add that any actual historians can perhaps comment in better detail. But if you want a resource to answer your question, and then some, this book is tailored to do so.