Doing this off the top of my head, so forgive me for lack of details, but the short answer is: Cars, fear, the Federal government, and local governments.

Longer answer is :

In the 1930s, while cars had been around for decades, American neighborhoods were still being built in traditional ways. Local shops in a neighborhood were normal and housing was relatively dense. But people were scared of the effect of "the street" on children, thinking they would be corrupted by being in the general public. People were also scared of children being hit by cars as they walked around on the streets.

So, during FDR's administration the Federal government wanted to encourage "safer" neighborhoods. They wanted more people owning homes instead of renting, more structured and cleaner environments for kids to grow and play in, a community oriented around public schools and institutions, and more places that looked like the ideas Le Corbusier and his followers were suggesting, that is to say, car oriented, detached buildings.

To achieve this, the FHA gave preferential loan rates to land developers who built neighborhoods that were strictly housing only, usually with a public school at the center of the neighborhood. Since back then it was expected that kids would walk to school, they wanted lighter traffic so the kids would be safer. This meant excluding businesses from the neighborhoods so that no one would have any reason to drive down those streets besides going to or leaving their house.

Even so, most neighborhoods in the 1930s had local grocers and shops. But by the 1950s that was changing. More regulations designed for safety created cul de sac neighborhoods. These were designed to lighten car traffic even more and make people drive slower to protect pedestrians and keep noise down.

The effect on businesses was that they could not survive in these low density environments. In order to get enough people to come to their shops, they had to provide a large amount of parking for cars and have easy access to streets and highways. If they stayed in a small building with no parking in the middle of a neighborhood, no one would have been able to come to their shops or even park.

These limitations were soon combined with local zoning regulations that were created on Le Cobusier's ideas of separating out every building from each other and, bam, you've got the American suburb.

And when local towns and cities realized after the 1960s that they could get way higher property taxes per capita by forbidding smaller, less expensive homes from being built and only allowing larger, more expensive, single family homes to be built in their communities, then it was set.

Large, single family homes on a legally mandated half acre lot with a large lawn in a neighborhood with minimal vehicle traffic were owned by fewer, wealthier families who went to better funded, less crowded schools. The neighborhoods got so big and so spread out that kids got driven to school in cars. And those families needed to buy stuff so they go in cars and drove a half a mile or more to the stores that had big enough parking lots to accommodate them.

To this day, if a land developer wants to build a neighborhood one of the first things they have to consider is how many houses per acre they can put on the land. Developers want more homes and families since they far more profitable. They can make more houses, build them cheaper, and are way more affordable for the public so they can sell more homes. But nearly always local government will want to have fewer homes, require larger houses, and bigger lots because they can bring in more taxes per capita. And, as far as I know, the FHA still gives preferential loans to suburban style developments.

All of that makes shops in a neighborhood financially impossible and are often straight up illegal due to zoning regulations that are now enforced cultural norms rather than the safety measures they once were.