The same way many unskilled workers are anti-immigration today, as they feel immigrants are taking their jobs.
Yes. One of the major arguments against slavery was that it wasn't fair for free labor to compete against unfree labor. Unsurprisingly, a lot of people who were strongly anti-slavery for economic reasons, were also racist. So for example, the state of Oregon, when it was granted statehood in 1859, explicitly banned black people (whether slave or free) from living, working, etc in Oregon.