As to New York City, at least, Robert Caro's The Power Broker: Robert Moses and the Fall of New York argues that part of the reason the city is both expensive and difficult to live in is because of mid-century (and especially post-war) construction priorities.

In brief, Tammany Hall (by which I mean, Democratic city mayors prior to La Guardia) left behind a decrepit and decaying physical plant. Roads were narrow and outward-flowing thoroughfares were nonexistent, subways were rickety and inadequate, public housing undreamt of. Under La Guardia, the city made great strides to eliminate the gaping deficits opened by Tammany-era mismanagement, but spent those savings on restorative work and improvements designed to benefit drivers, not commuters. This trend continued and worsened in the post-war boom, as the city built new bridges, highways, and tunnels, but left the subway substantially unimproved and unfunded.

All of this resulted in skyrocketing demand for parking spaces and car space within the city limits, which in turn is why rent on a parking space costs as much in Manhattan as a mortgage would in a midwestern suburb. It also means that the city's public transit (the MTA) has to charge nontrivial rates just to keep the trains running close to on time. Because previous generations failed to plan for the middle class, Caro argues, the middle class of today must fight harder and pay more for their share of city life.

Caro has not, as far as I know, been contradicted in this assessment, and his take on the defects in pre-modern city planning approaches gospel among New York City's political class.

This doesn't speak to real estate values, of course, but it might explain middle-class concerns in one of your identified cities.

You should better ask in /r/AskSocialScience (economy). Big and important cities are always more expensive than less important cities or the countryside.

I could try to answer for San Francisco. San Francisco is a city that has gone through multiple cycles of boom and bust economies, you can read more about the history of the city in a really fascinating book Imperial San Francisco by Gray Brechin. San Francisco has at times been an extremely desirable city to live in (during the boom times) and at others times has seen the same white flight phenomenon as other major cities, such as during the 1970's when the city famously became home to poorer immigrant communities (many of them Mexican-American) and members of various counter-culture groups.

Historically the city has always had a mix of the very well-to-do, and the less so, and in very close proximity to each other. That proximity is in fact one major issue. San Francisco is only approximately 7 miles by 7 miles, and bordered on three sides by water. It's geographic constraints have always been at the heart of land development in the city. That is coupled with policies set by the city in the past several decades to greatly restrict the development of new housing stock. In the 1970's and 1980's the city passed measures restricting the construction of new office space which sent a lot of jobs fleeing the city and reduced the population. But those same policies are now responsible for the serious housing shortage in the city and the exorbitant prices for rental housing and home purchases. San Francisco has some of the most byzantine and elaborate permitting procedures for new housing development. It is one of the few cities that does not allow simple permitting "by-right" meaning that even if the planning department fully approves a project in an area it is adequately zoned for, there is still a process by which environmental and neighborhood concerns can (and frequently do) bring new development to a halt. All of this has led to prices being so high for housing in the city as this has coincided with one of the city's cyclical booms - in this case the tech sector booms in the form of the dot-com bubble and the more recent tech boom in internet and startup companies.

There is a lot of complication in this issue, including many arguments made that San Francisco's housing problems are connected to the greater Bay Area and lack of regional housing and transportation policy and planning. Boom Magazine had a really good article on this subject if you are interested in a more in-depth read.

But to make a long story short, San Francisco has not always been "so expensive" and the cost of living in the city (primarily housing) is tied to cyclical boom and bust cycles in the regional economy as well as geographic constraints and restrictive city planning.