I understand that the work slaves did was terribly hard work. However I find it hard to believe that there were not unemployed people who would do any job they could find.
Yes, this was one of the principle arguments of the Free Soil party, an important predecessor of the Republican party that ran on the slogan of "Free Soil, Free Speech, Free Labor, and Free Men."
The converse argument was also made by people like John C. Calhoun, which is that an emancipated country would effectively result in a kind of wage slavery, and Calhoun believed that this would result in a kind of class warfare of the kind that Marx would describe decades later, although Calhoun reached the opposite conclusion about whether that would be a positive development or not.
Yes many did argue this. It was an especially contentious issue in cities where many slaves were given specialized training that would otherwise been a job for a free artisan. An example of this might be smiths.
Another issue was slaves who sold produce. In South Carolina, slaves were initially given substantial time to work small plots of land to create produce for their own consumption or sale. The sale of these goods (and slaves' acquisition of property) angered whites because slaves were now in competition with them rather than serving them economically. The opposition (largely) didn't come from plantation owners because they produced cash crops such as rice and indigo rather than the produce which slaves produced. Most of the opposition came from smaller farmers who produced crops for consumption and were therefore in competition with the slaves.