Because white Americans often believed African Americans were in various ways undesirable. For example, there were prevalent racist beliefs from Reconstruction through the Civil Rights era that: the men were hyper-sexual and wanted to seduce or rape white girls and women; they were predisposed to violence and crime in general; they were less intelligent; and so on. African Americans were also often regarded as labor competitors willing (in fact forced) to accept lower wages.
In the book "Buried in the Bitter Waters: the hidden history of racial cleansing in America", Elliot Jaspin investigates the stories of towns and counties that violently expelled their African American populations, and these stories show us reasons why white Americans often hated and feared African Americans. For example, in many of the cases, such as in Forsyth County, Georgia in 1912, there was an allegation that a black man had raped a white woman; in Polk County, Tennessee in 1894, it was black miners being brought in to work for lower wages than the incumbent white miners; these triggers led to wholesale expulsion of African Americans by threats and intimidation. Aside from a paranoia about rape, another fear was seduction: a major argument against school integration was that it would lead to interracial marriage, in which white parents would have to see their daughters take up with black men they regarded as uncouth.
After the Civil War, Southern states came up with new ways to force African Americans to remain in the subordinate position that most white people believed they belonged in. One example was vagrancy laws, under which African American men without proof of a paying job could be rounded up by police, jailed, and leased for forced labor. There was a prevalent belief that otherwise they would behave as lazy idlers with a penchant for crime. On this topic of forced convict labor, see "Worse Than Slavery".
Given this fear and hatred of African Americans, whites have in many places been determined to uphold white exclusivity in their lives. Hence discriminatory mortgage practices, whereby African Americans were not permitted to acquire mortgages on homes in predominantly white neighborhoods; hence "White Flight" in which white families from the Civil Rights era onward left city neighborhoods in droves for suburbs with more de facto segregated schools. On residential segregation as a response to the Civil Rights movement, see "White Flight: Atlanta and the making of modern conservatism".
In short, fear and hatred of black people has always reproduced itself in inequality and segregation that have led to more fear and hatred.
If you'll forgive me--and I hope the moderators will, in any event--this question has an implicit comparative dimension that is crying out for some address.
To some extent, this question relies on what is commonly called "American Exceptionalism", which is a perspective that is not exclusively held by Americans. Exceptionalism in this sense need not be positive: it simply holds that the United States is fundamentally unlike any other nation-state.
On certain claims, there's certainly something to this perspective. (Though historians are also adept at pointing out that every society and territory on the planet has some distinguishing particularity that makes it unlike everywhere else.) The history of higher education in the United States, for example, is extremely unlike anywhere else. The United States has a distinctively pluralistic understanding of citizenship that accommodates immigration (despite the current political climate) in some unique ways. And so on.
Racial ideology in the United States has some unusual historical dimensions, which is what motivates the OP to ask his question. How can a country that fought a punishing civil war over slavery and race have allowed a system of formal racial segregation to take hold so virulently in the century that followed? How could slavery have existed in a country whose Constitution so strongly enshrines Enlightenment ideas about equality and freedom? Why were practices of racial discrimination so persistent, all the way into the present?
The thing is, on most of these issues, while the United States has some distinctive or particular elements to its history, it is not actually very exceptional, especially in the time frame that the OP is most focused on. The French Revolution had to quickly confront the same contradiction between slavery and the Universal Rights of Man, and the answer was contradictory--but by Napoleon's time, resolved in favor of supporting slavery. France, England, Germany, Belgium, and Portugal all became involved in modern imperial rule over Africa in the late 19th Century; most of the same states were involved in imperial rule in Asia and the Middle East in the same time period. That imperialism was expressly, explicitly racist, including a number of cases of genocidal violence directed at non-Western societies or groups by white Europeans. European imperial armies in both World War I and World War II employed troops from their colonies, but treated them with as much racial contempt as white American leaders treated African-American soldiers. Africans, Asians and Middle Easterners who travelled to Europe before World War II were subject to racial segregation that was less formal but basically similar to Jim Crow in the United States.
It's only in the decades after World War II that some--and not all--people of non-European societies began to find that domestic European living conditions were less markedly racist than the United States, in particular, African-Americans who gravitated to Paris. There were always exceptions to this (London, for example) and Turks, Algerians, etc. did not find France or Germany warm or racially egalitarian.
So really in this question we are talking about a matter of degree, of small distinctions, not of major or dramatic divergence.