A pivotal moment in the history of historic preservation in the US was the passage of the National Historic Preservation Act of 1966. Jacqueline Kennedy advocated for its passage after the assassination of her husband. It was part of a legislative package that also established the National Endowment for the Humanities and the National Endowment for the Arts.

The First Lady had raised public awareness with her restoration of the White House. Subsequent to that, the construction of the Interstate Highway was often used by local communities as a means to demolish "blighted" urban areas. The result, however, was the destruction, using federal funds, of many valuable historic resources, without so much as a nod in the direction of their value. The NHPA of 1966, as amended, subsequently requires federal agencies to consider the effects of their actions on historic resources, and this has slowed demolition. The law also opens the door to tax credits and direct grants to encourage the preservation of significant historic resources. The law, then, represents a tidal shift in the mid 1960s towards the management of historic buildings.

Architecture and tastes did of course change. In San Francisco, for a long period of time from about the 1930s at the advent of streamlined architecture to about the 1970s when things started to be looked at again, the classic Victorian was seen as gaudy and busy. Today you see a lot of period facade renovations that stripped off the fine details of structures such as the ornate Call Building which was built in the 1890s, survived the quake, and was renovated in the 30s to be far more streamlined. Many SF single family residences were similarly converted to Brutalist or modern styles.

In the 1950s and 60s, you had the above issues plus other things at hand. Suburbanization and white flight led to the massive decline of central cities during that time, the "beautiful" buildings were getting to be 40 and 50 years old, many not well taken care of. The lack of parking for these buildings was critical--built in a time where few people had cars and most commuted by streetcars--these structures had no parking for customers, residents, and workers driving in...if any were left after cities turned into increasingly decentralized metropolitan areas. Thus razing a few old mixed-use street-facing buildings for single-use modern structures built off parking lots or structures was commonplace, and downtowns lost population.

As downtowns built modern high rises, many landowners engaged in "land banking", getting zoning entitlements, razing structures, paying less in property taxes, and selling parking spaces until they too would build a highrise.

The structures that remained were by in large scale obsolete until preservationists found new uses for them in the 1980s-90s--the exact time depends on where you're talking about as the historic preservation varies in popularity across the country. Manufacturing and warehousing moved to spacious freeway-oriented locales where rail was less important, leaving vacant loft buildings. Beautiful old single family homes have crappy floorplans compared to open modern homes, suburbanites chose space and room for parking so the only people that ended up living in central cities were there because they had no choice. Accordingly, the impoverished conditions in central cities from 1970s up until after the crack epidemic of the 1980s (or later) led to many structures' demise or further deterioration. Arsons were common in Phoenix and New York City as property owners sought cheap ways to bail themselves out of the liabilities of a standing structure.

Urban renewal was also all the rage back in the 1960s. These would be concerted efforts by city and state governments to retrofit central cities for modern amenities, displacing large amounts of the remaining population who tended to be impoverished minorities. Cities razed block after block of their vacant and crumbling downtowns for large scale auto-oriented projects like sprawling Modernist government centers, convention centers, sports arenas, stadiums, shopping malls, and freeways. Empire State Plaza in Albany is by far the worst example, imho. In SF, vast expanses of the Western Addition, dating from the 1880s and a largely black neighborhood, (i've heard harlem of the west) were destroyed and replaced with wider boulevards and newer residences with an abundance of off street parking. Old buildings were simply old and in the way, and there was little in the national mindset for historic preservation until pioneers like Jane Jacobs and others pushed back against the tide.

The freeway revolts in the 1970s put a stop to a lot of things, but in Phoenix's case for example urban renewal and lack of interest in historic preservation let it continue unabated until the mid 1990s.

source: my own head, i've been following urban design and historic preservation movements in a variety of cities in my spare time for the last 10 years.

Even if there was a public outcry—as there was with Penn Station—there simply wasn't any mechanism to force the owners of buildings to preserve them. Only the 1978 Penn Central case, which saved Grand Central Terminal, finally allowed municipalities to put up barriers to demolition with any legal certainty.