I am not trying to be offensive or racist in anyway, i am just curious. I mean, in those times it is not like the majority of the white population were releasing them because of benevolance, so in that case why did they not just forcibly remove them, I do not really see what the point was of allowing them to stay in the United States if they did not even like them to begin with. Also how come the African American population just not leave the country since they were being discriminated against so much after slavery.
This is a situation where you need questions more than you need an answer (not that there is a singular answer to this).
Who would be sending the freed slaves to Africa? The American Colonization Society, which funded the emigration of free Blacks to settle/found Liberia was the pet project of a few wealthy men. While it did get state and federal support, it was always minimal and the ACS ultimately ended up only funding the emigration of a few thousand settlers. A paltry number compared to the millions enslaved in the US.
Why would they want to go? The slave trade was ended in the US, per the Constitution, in 1808. While some smuggling did occur almost up until the Civil War itself the supermajority of Black people in the US had had generations to form a culture unique, independent, and cut-off from Africa. This was helped along by efforts of slavers to eradicate African traditions and language. It's why "African-American" is a distinct culture unto itself, within the larger context of American culture. Why would a freedman opt to board a ship to a continent he's never seen and barely knows anything about, leaving behind everything he does know? Even the Garveyite Back to Africa movements of the early 20th Century were stymied by this. This is on top of the hurdle of being able to afford picking up stakes and relocating a few thousand miles away to a continent on the other side of an ocean.
Why weren't they forced? Even when the slave trade was in full swing it was abhorred. Thomas Jefferson, in his rough draft of the Declaration of Independence, had a passage condemning the trade, if not slavery itself. Keep in mind that Jefferson was one of the largest slave-owners in Virginia. The abolition of the trade, but not the practice of slavery, which was written into the Constitution was also reflective of the bifurcated approach to the "peculiar institution." Again, it was the slave trade, not slavery itself which was more generally agreed upon as a distinct evil. Even the Confederacy opted to uphold the ban on slave trading in part because they did not want that stigma attached to them (among other reasons, I wrote a past post on this). Forcibly removing the 4 million freed slaves would not only be a morally unteneable proposition, but also a logistical impossibility; that was almost 13% of the US population at the time.
Why would White people want them to go? Emancipation, particularly after Reconstruction ended, actually meant very little changed for the Black population. The 1860 Census counts about 500K free Blacks in the US, concentrated mostly in the Upper South, but generally spread across the nation. The 4M soon-to-be former slaves, in contrast, were overwhelmingly concentrated in the South (giant PDF map). It wouldn't be until the Great Migration that large numbers of African-Americans moved out of the South and into the industrial cities of the Midwest and Northeast. The overwhelming majority of Black people were in the South, were they endured conditions not far removed from slavery as indebted and intimidated sharecroppers and laborers (if they were not still living in actual slavery). The White Southern elites still had a cheap labor pool to draw on and all Southern Whites had a permanent underclass to distinguish themselves from. It's not that Whites "did not even like" Blacks, it's that they saw them as inferior; the paternalism that characterized Southern slavery continued on after emancipation.