Railways began as private enterprises with limited funds, and typically were thought of initially as completing transportation links (to a major harbor or river port) rather than as a new transportation network. Thus, most major cities ended up with multiple railway terminals as each railway sought the best terminal it could arrange for.
Built-up cities were not anxious to have the noisy, smoky railways intrude into the existing fine-grained urban fabric, and available station sites were almost always outside the heart of the existing city. Thus London ends up with the row of railway stations, for lines serving the North of England, along the northern edge of the 19th century city, and a similar row along the south bank of the Thames—or just a few yards beyond the north bank—for lines from south of London. Paris ends up with nine terminals, Berlin a half-dozen, Boston eight, Chicago six, Rome five. The situation was similar in nearly all European capitals, as well as Moscow, Tokyo, Bombay, and Melbourne. New York’s lines to the north had to settle for building a new terminal (Grand Central) all the way out at 42nd Street, while several of New York’s terminals weren’t even on Manhattan Island.
As soon as railways became major passenger carriers, the problems of multiple terminals were recognized, but few cities were able to muster the will and money for enormous viaducts through dense districts to allow a single station or through-running of services. An 1846 Royal Commission had conceded the desirability of connecting London’s terminals, but nonetheless recommended against undertaking such an enormous project. Instead, the initial lines of the London Underground eventually served the purpose of moving travelers into the center of London, and to other terminals to continue their journeys. Berlin was a notable exception, completing the seven-mile-long, four-track stadtbahn through the city, mostly beside the Spree, in 1882.
A 1915 proposal for through-routing Chicago’s steam railroads gives a contemporary overview of such proposals. I believe Brussels and Madrid eventually connected their two main stations with a north-south trunk, while Tokyo and some other cities developed a ring railway connecting various terminals. Melbourne and Sydney dug distribution rings around their central business districts, while Munich and Philadelphia dug tunnels from one station to another in the 20th century to allow through running of suburban services; perhaps the most impressive such efforts were Paris’s RER and London’s soon-to-be Crossrail.