It's important to consider how the relative damage that, say, a bomber crashing into a neighbourhood would compare to the damage of an actual bomb dropping.

In general, planes being shot down and crashing into residential areas or factories was greatly overshadowed by the damage caused by the bombings delivered by said planes.

This combined with the fact that of all the places a plane could crash after being shot down, the vast majority of it would be areas other than heavily populated areas.

It isn't as if every plane just went straight down over the patch of land that it happened to be over at the exact moment of being shot down. Even losing control directly over a city, the plane could easily come down outside of the city.

In general, planes crashing into houses and factories were the least of the list of worries when enemy planes show up in force over head. It was just an inherent part of the risk.

As for the defending side, shooting down an enemy plane overhead could very well bring it down on them, or on the very people and areas they were protecting. But the comparable damage of letting those planes continue their missions unhindered is quite more frightening.

/u/AsiaExpert gives the Best Answer. This did happen occasionally, though as indicated any such events would typically be lost amongst the destruction of the bombing.

One event I know of is that the first civilian casualties in England during WW2 were caused by a Heinkel He111 that crashed in Clacton-on-Sea on May 1st 1940:

http://ww2today.com/1st-may-1940-first-civilian-deaths-in-england