It's a combination of three things: competitive overbuilding, changes in the industrial landscape, and the growth in intercity trucking.

Railroading was a cutthroat business in the late 19th century, with a certain irrational exuberance in the expansion decisions, meaning that lines were extended into towns and areas where they could only be profitable if subsequent growth was unbelievably great. Sometimes extensions were done to undercut competitors; sometimes on faulty prognostication about the promise of a farming or mining district. Often the promoters made their money not on actual transportation, but by selling vast numbers of shares and forming corrupt construction companies that overcharged the shareholders for building the line, such that most railroads went into bankruptcy within a decade.

Second, the industrial landscape has changed greatly in the last century: entire areas of agricultural production have fallen into disuse or low-density grazing use, lumbering has changed to managed forests, meatpacking now takes place adjacent to ranches or feedlots, coal mines have closed entirely or become large strip-mining operations, industries like iron and steel production have consolidated into a few large operations or moved overseas. Manufacturing was able to relocate from dense cities with multistory factories and warehouses to low-cost rural areas where land and labor are less expensive, including relocating overseas.

Some of this transformation preceded the growth in intercity trucking—and the better highways they made necessary—but some of it was made possible by this innovation. The one-dimensional nature of trains means that trucking will always be more agile on moving things from dispersed origins to dispersed destinations. The result has been that railroads, like marine transport before it, has come to focus on moving large loads of heavy things—coal, oil, chemicals, lumber, grain, groups of freight containers arriving at ocean ports—from one location to another a thousand miles away. Loads such as lumber or grain are consolidated from trucks arriving from multiple origins; loads such as finished goods (in ocean-going containers) or gasoline are distributed to multiple destinations within 500 miles by trucks.

This tendency of railroads to focus entirely on big shippers moving tons and tons of stuff at one time has been intensified by the demand from Wall Street for those publicly traded companies to constantly increase profitability. The main way this has been done—at least, in the short term—is by precision scheduled railroading, which generally means focusing on the big customers, closing classification yards and laying off employees other than those needed to move a few long, heavy trains from origin to destination.

Were the mothballed rail lines a result of overbuilding due to a stock market "rail bubble?" or merely a result of subsequent competition from automobiles?

Both;

#Railed Roads

The United States was poised to utilize and develop the industrial locomotives being invented in Europe in the early 1800’s. One reason the USA doesn’t have as much canal infrastructure as Europe (especially the UK) is that steam-powered alternatives were coming around during the nations formative years. America offered fuel and raw resources in plenty; historical accounts often describe it as an unimaginable bounty, totally untapped by the native population.

Railed roads were nothing new, mines had used them for centuries and farmers had used wooden railed roads to move ‘dandy wagons’ (a train of wagons pulled by a team of horses) of produce/grain to city centers. In the UK these roads were owned by an entity, usually municipal, that was forbidden to also operate on the roadway, least they become a monopoly. Farmers bought the ‘right of way’ to use the rails when harvest came, they even developed token systems so a farmer could PROVE he had bought the right of way should he encounter someone else in his way on the rails.

#Tom the tank engine

High pressure steam changed everything. Instead of teams of horses being needed, as well as replacements every so-often since horses tire, an iron steam kettle could move the same load in half the time. Took a few decades but while the UK developed what would become Tom the tank engine in the States we developed a roughed locomotive platform for the wobbly railroads the pioneers were blazing across the countryside. There were no shortage of people wanting in on the bounty waiting to be tapped, industrialists were happy to seize the opportunity. Banking families formed from the few cut-throats that had made a fortune during the countries early days, many of these families got involved in building railroads.

#American steam

The nation experienced a series of railroad booms and busts from the civil-war to WW1. The cycles typically went something like this; new resource/region becomes available, US government encourages growth by providing railroads with free land to motivate them to build lines, lines are laid—often never finished should funding dry up, various bankers and malcontents get involved (as they always do where-ever theres money) and the speculation enters ‘full steam.’ But railroads cost a fortune to maintain—-steam engines themselves are a lecture on routine maintenance but the rails themselves requires lots of work that people don’t think about. Shares in dubious projects were issued and sold, guess what happened to many the investors money?

Like all speculative bubbles they eventually burst. Pre-auto the biggest cause of railroads closing down were often the railroads poor management and finances. Still, the rails grew as the surviving companies bought up defunct railroads engines, rolling stocks and even railways. America was still a growing nation, trains still rolled right into the middle of ‘downtown’ where armies of people shuffled the contents out and to the stores while others brought wagons of raw materials to ship on out. A tap-dancing quartette of industry.

#Automobiles

Railroads have never been the best or only mode of transportation. Water is still cheaper per ton to ship on (been that way since the Roman empire) but rail roads were better for passengers and many forms of cargo for several decades. But now the internal combustion engine was taking off—it literally allowed people to fly. When autos became prolific enough (the technology had matured enough) it began to eat into the passenger services first. Lots of places in America has ‘inter-urban’ trains, typically a modified passenger car with an electric motor installed for propulsion. These were the highways that connected city centers with smaller town centers for several decades (post lightbulb but pre-great depression.) They were the first railed victims claimed by the auto-army.

Trollies, electric inter-urban passenger services, were bought up and paved over in favor of the automobiles buses. Diesel engines really fueled this transition, as did the Big Three auto companies directly targeting them. Lots of lost railway in the USA is still there, buried under main-street.

#Alro Guthrie as his eulogy for magic carpets made of steel

No one found a more beautiful way to describe what the railroads HAD been and HAD become than Arlo Guthrie’s The City of New Orleans. It describes what passenger rail service once was in America, juxtaposed with the ever changing landscape. Eisenhower's interstate system deserves an honorable mention since they allowed larger trucks to routinely cross terrain once reserved for freight engines. Arlo understood what was really happening, just as Mark Twain had when he witnessed the demise of his beloved paddle-boats fifty years earlier.

By the 1970’s America passenger rail service was in a bankrupted state. The remaining passenger lines were all merged together into a government subsidized American-Track; Amtrack. It was a rocky start; rolling stock so old and rusty it barely meet safety requirements. Over the decades the service would salvage and operate on the most profitable lines, often allowing freight provides to use the same rails during off-peak hours as a way to make money while still protecting a vital bit of infrastructure. They have a respectable fleet these days, typically more popular in densely populated areas.

#Railed roads today

The reason we see other nations with more sophisticated rail networks than the US is simple; the evolution of rail service in their countries was vasty different in how it was coordinated. Same reason there aren’t any grand canal systems in the States; we did it differently. Heres a few historical bits I skipped past but will mention here;

• The US COULD have electrified the lines back in the 1970’s—it was discussed and dismissed as expensive. Pretty much everywhere else in the world electrified and are better off because of it. America can still electrify the lines and offer a great boon to freight providers, they’re pretty much the only railroad companies left and they make profits when the economy is running normally.

• Much of the run-down rail you see these days are obsolete lines; old service lines for companies that are no longer there, junctions that are no longer needed since freight traffic has shifted as technology has changed, or are owned by a private freight line that probably uses it only as a side-track to store rolling stock on. Oh, and a handful of ‘heritage lines’ that are tourist attractions (technically it is a form of passenger service.)

• The vast distances between cities is often cited as a reason why we didn’t get much high-speed bullet trains. Its a bit of a lie, if they’d built the railroads like a high speed rail line you’d see profitable service on it. Problem is these lines cost a fortune to build and maintain and people have a variety of transport options these days. If an area, say California, actually built a properly isolated high-speed line you’d see it provide utility in the areas connected to it. Very much a NIMBY issue. Luckily, European nations dived into such investments and we see the benefits they can provide.

#TLDR

Ultimately the rise and fall of the American golden age of rail is the result of greedy mismanagement and an over-reliance on freight as a revenue generator. Lots of cities have passenger light-rails still, but most Americans only ever encounter the freight service lines because thats all they feel they can make money on anymore.

Autos weren’t really the death knell for the railways that history makes them out to be. Autos allowed more Americans the ability to travel, the fleets of semi-trucks running on a ribbon of freshly poured concrete killed most investment in the iron railed ways. Want to blame someone for the sorry state of American rail? Its the ones in power who spent the investment on alternatives, for better of worst, plain and simple.

#The Future

Rails ain’t going anywhere. China didn’t build the worlds largest rail network in a decade because they have a soft-spot for Tom the tank engine. Every developed nation should be utilizing its railed roads effectively. The USA could electrify the lines within a decade, usher in a new era; the 21st century electrified system. Freight would be cheaper and cleaner to ship; this has always been the saving grace of freight lines—lots optimize that ratio. Passenger service is still popular in Metro areas, usually run by an independent organization. These, as well as subways, can be rejuvenated and offer another century of trouble-free productivity.

There are very few practical limitations to realizing this vision; its a simple infrastructure project thats been tested and proven the world-over dozens of times now. Trains aren’t dead in the USA, they’re just running a bit late catching up to us here in the 21st century.