There were quite a few people who were pushing the "Back to Africa" movement. In the North there were groups, such as the American Colonization Society, which were made up of religious leaders and abolitionists who were interested in helping American born Africans move back to Africa. The ACS helped found the colony/country of Liberia, whose capital (Monrovia) was named for the President at the time. There were also southerners who were part of the ACS. They supported sending freedmen "back" to Africa because they feared that freed slaves could encourage slaves to revolt, or they could participate in a riot/revolt themselves. It was a unique coalition of folks.

After the Civil War, the back to Africa movement became more of a black cause. Due to hostility in the South (Jim Crow, KKK, lynchings), many freed slaves emigrated to Africa. At the same time, many refused to leave the US because Africa wasn't their homeland. They were between 100-300 years removed from their ancestors that came from Africa. By the end of the 19th century, the back to Africa movement had died down, but throughout the early 20th century (Marcus Garvey) there were still pushes every now and again, but they didn't really pick up steam.

As far as getting rid of the blacks when they were freed, the South still needed labor. Even though blacks were freed at the end of the Civil War, it took less than a decade for the white ruling class to reestablish their power and to enslave the blacks again through the sharecropping system and Jim Crow. The free labor of slavery was replaced by the deeply discounted labor of sharecropping. In 1870 blacks made up over 35% of the population of the South (compared to just under 20% today). If they shipped out 7 of 20 people, their labor force would have been decimated.