While it may not be the pure historical answer you're looking for, I can speak about this in terms of criminology.

The problem you're talking about, crime in poor inner city districts, relates to the Chicago or Ecological school on criminology and in particular to the subtheory of Social Disorganization. This theory links crime to spatial locations within the city, rather than the inhabitants.

Shaw and McKay (Juvenile Delinquency in Urban Areas, 1942) ran a comprehensive study of Chicago over the decades to study crime patterns, and the areas they found be criminogenic did not significantly change, despite near complete population replacement in these areas from Western European immigrants at the start to Eastern European and African-American immigrants by the end. Their argument is based on the ideas that crime is a response to social conditions caused by the design of cities. These areas have little governmental or societal structure, which results in criminogenic neighborhoods.

While they ran this study in Chicago, I have no doubt that applying the concentric model of cities to NY would place Harlem in Zone 2, the Transitional Zone. Because the model is based on transportation and city growth, Harlem did not always exist in that zone, but does now.

Because these areas are generally poor, with low property values, they are in US history occupied generally by immigrants or by African-Americans, who even today live far below national average income levels.

Other authors you might look into include Edwin Sutherland, Robert Park or Edward Burgess, the last two being the ones who created the concentric model in 1925. More modern takes on the theory should start with Robert Sampson.