I know the narrow gauge had been used first and was spreading by the introduction of Brunel's 7 foot gauge in 1840, but why did it die? It was cheaper to maintain than narrow gauge with lateral cross sleepers, and it meant that an economy of scale was in place where greater numbers of people and goods could be carried. Not only that, there were mixed gauge tracks available and that were used, yet 7 foot track was almost never used following the 1870s.
First point to make is that it was inevitable that one gauge would eventually predominate. The varied gauge measures you reference in the early 1840s existed because railroads were, at this time, a regional system of transport and supply. Uniformity was not necessary. I understand this is not your question, but it's important to weigh the question considering that it seems inevitable that one system would eventually win out. One amazing episode that highlights this (and which explains why 7 foot track dies out "following the 1870s" as you say) was the standardization of tracks during and immediately following the Civil War. Standardization was mandated by law in the North, and all former Confederate tracks were standardized after the war in a tremendous effort over a very short time span (much of it in an astonishing two day period).
As to the heart of your question, the answer is less clear (at least to me). I think there are a number of factors, including:
Standard track was preferred by locomotive and railcar manufacturers, so railroads that build on standard had more options to select from.
Broad gauge required a larger footprint, generally. Even a few feet mattered when the federal government had to pay for the taking of easements across existing property.
The transcontinental railroad used standard gauge. This encouraged other railroads to mimic them to make it easy to connect.
I am sure there is a better technical answer out there that cuts deeper at the heart of your question (basically, why standard gauge was seen as "superior"). Unfortunately, I don't know. It does appear that both narrow and broad gauges were falling into disfavor by the mid 1800s and much of the dispute between gauge sizes came down to a matter of inches rather than feet (rival gauges like the Ohio gauge were just a couple inches different). Maybe standard gauge was merely a happy medium, or maybe there are deeper mechanical advantages when constructing the types of railcars that were used at the time.
But "broad gauge" in the sense of a gauge wider than Stephenson's "standard gauge" did not die out, even in the UK. Irish standard gauge was, and is, wider than "standard gauge", as is the Spanish and Russian standard gauges.