So, why did rail-based public transport fail in North America?

There's a few reasons why, all of them interrelated. (This post borrows from what I've written elsewhere on my own subreddit, /r/lostsubways, and my forthcoming manuscript, The Lost Subways of North America.)

I'll focus on the reasons why rail mass transit was ultimately shut down in Detroit, because it's what I have on hand right now.

We'll start by comparing Detroit's streetcar system in 1905, with what it looked like before its final closure in the 1950s. Seems like a big reduction, doesn't it? There's three big factors that led it to decay like this.

1. Progressive reformers and anti-monopoly agitation of the early 20th century. Like most cities which grew up in the early 20th century, Detroit used to have a truly massive streetcar system, and all of it was run by an evil monopoly known as the Detroit United Railway. In short, you could get anywhere on the D.U.R. or one of its subsidiaries, with trains that were frequent and fast. Today, a transit line is considered frequent if a bus or streetcar comes every 10 minutes or less; in the early 20th century, this was considered a bare minimum level of service in urban areas and even in inner-ring suburbs. This level of service, though, is only financially feasible in the absence of alternatives. At the turn of the 20th century, if you didn't want to walk, you just didn't have any plan B. Cars were still expensive toys for the rich. Buses hadn't been invented yet. Horses were slow, and of course, you had to feed and water them. This gave the D.U.R. a monopoly over transport, and with this monopoly came a tremendous amount of power. All of this meant that the D.U.R. was a target for progressive reformers who wanted the Government to break the stranglehold of the D.U.R. by seizing control of the railroad. Ultimately, the City of Detroit bought out the D.U.R.'s local lines in 1922.

2. Better motor vehicles. The D.U.R. monopoly was fragile, as competition from cars, buses and trucks made it uneconomical to operate many of the train lines that ran deep into the suburbs, and much of it was converted from rail to bus. In Detroit, as in most cities, the streetcar networks reached its zenith around World War I and declined thereafter. Technology is a strong factor in this. Point-to-point transportation for individuals was better handled by the automobile. For lightly-used transit lines, buses could do the job just as well, and at lower cost than streetcars, because there were no overhead wires or tracks to maintain. Similarly, for light freight loads, a truck can go direct from a railhead or a port to its ultimate destination. Rail-based surface transit works best when trains have their own trackways or lanes, they run long trains, and have frequent service, like with the modern Green Line in Boston. But one-car streetcar service with no dedicated lanes, as in Detroit, is just a bus that can't change lanes.

3. Mass transit was a business. Today, we consider mass transit to be a public service, but in the early 20th century that wasn't the case. Mass transit was supposed to make money, and when a line wasn't profitable, the general expectation was that the operator would shut it down or convert it to bus, rather than continue losing money. It took until after World War II for American cities to conclude that transit operations had to be subsidized, long after governments had taken over operations. In Detroit's case, this means that rail lines continued to be shut down for most of the 20th century by the city government and replaced with bus lines. Tellingly, the system was largely shut down before large-scale freeway construction became a thing - the first freeway in Detroit, the Davison Freeway, wasn't built until 1942, long after the streetcar system had entered its terminal decline.

Alongside the general reasons mentioned by u/fiftythreestudio for the failure of rail-based public transit, the railway companies simply weren't interested in lobbying against the automobile until it was too late.

The first reason was that the rail companies were all doing badly around the time the automobile rose in prominence. The 1900s and 1910s brought anti-trust action that broke up many of the monopolies, and economic downturns such as the Panic of 1907 and the Great Depression also affected the rail companies significantly. The railways that didn't enter receivership were struggling to keep afloat (they also abandoned many rail lines, including less profitable passenger services) , and the more successful companies were trying to acquire and consolidate railroads. Lobbying against the automobile was very much a secondary priority.

Another was reason is that the railway/public transit lobbies didn't always see the automobile and federal support for roads as directly opposed to or competing with public transit. In the case of the interstate system, the public transit lobby actually supported the Interstate system. In the years prior to the Federal Highway Act of 1956 that would form the Interstate system, the American Transit Association, which represented many public transit operations both public and private, advocated the position that the highways weren't direct competitors with public transport systems. Instead of directly opposing the interstates, they instead lobbied for the freeways to accommodate public transportation infrastructure, such as light rail on the highway medians and dedicated bus lanes. To some extent, they achieved their goals with funding for certain mass transit projects like the Congress Line in between the Eisenhower Expressway, the Lake-Dan Ryan Line between the Dan Ryan Expressway, and a dedicated bus lane along the San Bernardino Freeway in LA. The ATA also pushed for a public transit exemption to the gas tax, which they also got in the FHA.

Sources:

Leonard, W.N. The Journal of Economic History , May, 1949, Vol. 9, No. 1 (May, 1949), pp. 1-24

Schwartz, G.T. (1976). URBAN FREEWAYS AND THE INTERSTATE SYSTEM. Southern California Law Review, 49.

I'm just sticking my neck out but there's an aspect of OP that it seems worth commenting on. Mostly the idea of a "massive railroad industry" in the late 19th century United States. It's probably worth clarifying that there was, but railroad companies in that context are not really the same thing as rail-based urban mass transit. Which is to say, the Pennsylvania Railroad was one of the largest corporations in the US by the late 19th century, but they weren't, say, operating trolleys in Philadelphia. It's even more so for the transcontinental railroads of the late 19th century - Union Pacific still exists after various corporate permutations, but it's always been a long-distance railway concern, and not operating things like subways or trolleys.

Along those lines, it's worth pointing out that freight rail traffic was and remains a huge business in North America - it's passenger rail that declined to the point that passenger rail services were consolidated into Amtrak in 1971. But there are still seven Class I long distance railroads and over 630 Class II and III medium and short distance railroads, and freight rail is still the single biggest form of transport by ton-mile in the United States. US railways are bigger than they've ever been.

Hopefully someone else can comment a bit on this a bit more, but another factor in the shift of railway companies from providing passenger services is the ending of Railway Post Office services on passenger lines (ie, having post office cars that sorted and delivered mail by rail). The shift from this to highway freight delivery began in the 1940s, and the last railway post office services ended in 1977. My understanding is that this was a significant factor that subsidized passenger rail services, and its decline along with the other factors mentioned helped make passenger rail far less profitable to operate in the United States in the md-20th century.

Can you clarify? Who has ever claimed that the "car lobby" influenced policies toward public transport?

Ford and GM put their public relations efforts behind promoting better, faster, safer highways, including urban superhighways, but I've never come across anything from them denigrating intercity passenger rail or urban transit.