James Gordon Bennett, Jr, publisher of New York Herald, famously said, when he leased instead of purchased land for his new headquarter in Herald Sqaure, "Thirty years from now the Herald will be in Harlem, and I’ll be in hell!" What stopped that migration?
By the time Bennett said this (the mid-1890s), the city had seen several Northward pushes at increasingly rapid rates. Often, these coincided with the locations of both the newspapers and the main entertainment district. Starting in lower Manhattan on Park Row, the fashionable areas steadily pushed North, settling on Astor Place (until the Astor Place Riots), Union Square (a center of early film, plus the nearby Yiddish Rialto), the Tenderloin (which included Tinpan Alley), Herald Square, and finally Longacre Square (renamed Times Square in 1904). Some papers and theaters moved farther uptown, but for the most part the major commercial expansion stopped around Times Square and midtown, as both the Upper East Side and Upper West Side had prominent residential districts. Skyscraper construction was well-suited to midtown, as the bedrock is close to the surface, providing a stable base for taller buildings.
Interestingly, Bennett was not the only person who foresaw Northern expansion. One of my favorite examples of this was the New York, Westchester and Boston Railway, which didn't push to maintain a connection to Manhattan (a la the New York Central route to Grand Central), instead choosing to build a terminal at 132nd Street and Willis Avenue. The developers of the Railway thought that Northward expansion would push the city's economic center into the Bronx; that wasn't the only thing they got wrong, as the company later went bankrupt and was bought by the city, forming the Dyre Avenue line (No. 5 train) of the subway system.
N.B: I am writing this on my phone, but if you want sources or have any other questions, feel free to ask, as I'll be at my computer later.