Slavery affected poor whites tremendously. It was the institution of slave labor and poor education that kept poor whites from making a living at all. Surprisingly enough, the relation between white slave owners and the poor whites were not very good. Here's a quote by John T. Campbell:

In the past, white men have hated white men quite as much as some of them hate the Negro, and have vented their hatred with as much savagery as they ever have against the Negro. The best educated people have the least race prejudice. In the United States the poor white were encouraged to hate the Negroes because they could then be used to help hold the Negroes in slavery. The Negroes were taught to show contempt for the poor white because this would increase the hatred between them and each side could be used by the master to control the other. The real interest of the poor whites and the Negroes were the same, that of resisting the oppression of the master class. But ignorance stood in the way. This race hatred was at first used to perpetuate white supremacy in politics in the South. The poor whites are almost injured by it as are the Negroes.

So, by this quote, we can definitely infer that even back then, white slave owners knew slave labor negatively affected poor whites but because of racial hatred and ignorance, they actually helped maintain slavery for so long, and it was in the white slave owners' best interest to create tension between the negroes and the poor whites. As a result, the negroes became even more of a scapegoat than before and the poor whites never progressed politically or economically.

Edit: I want to add that the enactment of the Poll Tax was not entirely targeted towards negroes, who obviously didn't have any money, but poor whites who also had no money. This prevented both negroes and poor whites, who were lower class citizens, from making any political changes. So, we can see that even though issues back then were rooted in race, there was also class warfare going on as well.

Older thread on the price of slaves here.

Those figures are adjusted for inflation only. If you consider relative buying power to today, buying a slave was somewhat like buying a house. It was not a cheap thing.

Slavery was a competing source of labor, and thus inevitably made poor whites less well off. However, the competition wasn't as free as one might imagine. If humans are treated as industrial inputs, and traded as such, prices will tend to adjust such that the marginal return to capital is equalized. This means that any large profits from slaves will be eaten away as prices get bid up. A study by Alfred Conrad and John Meyer found that the return on slaves was comparable to other investments at the time, such as railroad bonds. When we consider that slaves are being traded and priced in a manner similar to industrial equipment, it's not really free, so while they can lower wages through increased supply, they can't drive them down forever. Unfortunately, I can't readily find a source of wage distributions from the time.

There's a sort of second-order downward pressure on wages as well, depending on which interpretation of the southern economy you believe. Ralph Anderson and Robert Gallman argue that the fixed costs of slave ownership discouraged specialization and trade, and hence urbanization, and ultimately economic growth. This is because while slaves aren't free, once you own them, the marginal cost of doing something with them is near-free, as most of the costs of keeping slaves are constant. Plantation owners would prefer to do as many things in-house as possible, rather than trade for them, and thus diversified into many operations, albeit inefficiently. Anderson and Gallman argue that this economic organization stalled industrialization and new, more skilled job opportunities in the South.

I can't answer this question fully, but having taken a class on American Reconstruction I can shed some light on the dynamic between poor Southern whites and black slaves. You may find Eric Foner's Reconstruction: America's Unfinished Revolution relevant to this question. It is Foner's discussion of Andrew Johnson that I cite below.

Most histories of this period divide the white South into two principal economic classes: non-slaveholding yeomanry of the Southern upcountry, and slaveholding planters of the plantation belt. America's 17th President, Andrew Johnson, hailed from a family of yeoman farmers in Columbia, Tennessee. In such upcountry regions, conditions were not suitable for the large scale plantation agriculture that existed in the Southern lowlands. Thus, the economies of Southern piedmont regions consisted mostly of subsistence farming and cabin industry. Johnson's origins in this environment largely informed his policies on race and economics in the South when he came to office. He resented slavery because the institution promised enslaved blacks a more stable lifestyle than lower class whites. In Johnson's view, slavery served as a tool of political and economic oppression toward the Southern yeomanry, a class that could not afford slaves. In effect, Johnson believed that slavery upset a natural racial hierarchy by condemning poor whites to a lower social status than enslaved blacks. Of course, Johnson's beliefs do not necessarily hold true for the entire Southern yeomanry, yet they do suggest that lower class white Southerners may have viewed slavery as an institution counter to their economic interests.

Concerning wage competition, the postbellum Black Codes of the South may be of interest to you. The Black Codes represent the attempts of Southern legislatures to prevent the emergence of a free labor society in the South. The codes controlled the black labor force by imposing strict laws that limited freedmen's rights to move and negotiate fairly. Notably, codes also dictated that planters could not use competitive wages as a tool to lure black laborers from other plantations. With artificially suppressed wages imposed on blacks, I imagine that a poor white laborer seeking fair wages would have had a difficult time procuring a job as a sharecropper. In fact, this labor surplus largely contributed to the emergence of racial violence in the South in the form of the Ku Klux Klan (and other organizations). Klansmen used terror as a tool to further control black labor in the South and make the economic climate more favorable for white sharecroppers and planters.

I know this answer dealt with the postbellum rather than antebellum period, but hopefully it provided some useful information on the racial dynamic within the Southern economy.

Yes, the existence of slavery drove down the quality of life for poor white workers. "Wages" is a hard word to apply to this because people were generally farmers and worked for themselves. In a country of inexpensive land, why work for someone else when you could move somewhere and work for yourself? But, poor whites in the South would have been selling their goods in the same market as slave holders, who had a comparative advantage and could sell goods for a lower price and still live well. The price was driven down and poor whites who did not own a slave were unable to live well. Buying a slave was buying a place in society.

It's also been demonstrated that slavery gave the South the absolute advantage in growing cotton and sugar. Slaveholders would turn this money into buying more slaves rather than industry. At the same time, the people who loaned money to the South lived in the North and re-invested the capital they earned into industry there, rather than the South. All of this means that there was little factory work in the South of poor whites.

This left only a hand full of jobs to wage workers. In the port of New Orleans, a person could work in the docks loaning and unloading ships. This is where you would see slavery driving down wages for a white person who was willing to work for a wage (again, there weren't that many). Some of this work was done by slaves, free blacks, or slaves rented out to work for the day.

Because of this, many poor whites moved to Northern states like Illinois and Indiana. This encouraged anti-slavery sentiment there but these people where still steeped in racist ideology from birth.

Sources:

Walter Johnson's Soul by Soul and River of Dark Dreams John Mack Faraghe's Sugar Creek

I am surprised nobody has mentioned the most classic piece ever written about reconstruction: Black Reconstruction in America by W. E. B. Du Bois. I believe he was the first to put first the theory that poor whites were duped into supporting slavery and later white supremacy even though it was against their best interests. Excellent read.

Follow up ?: It's my understanding that the Civil War was seen as a "Rich Man's war" by many poor southern whites. How widespread was this and was there any resistance to getting involved in the war among poor southern whites?

In some of the more marginal (agriculturally wise) regions on the South, white smallholders were so poor and so disenfranchised that they supported the Union during the war.

References: http://www.freestateofwinston.org/
http://www.1stalabamacavalryusv.com/

Demographically, areas like Winston were more similar to West Virginia. That is, very remote, no plantations, with poor pioneers living there without any slaves. They weren't exposed to the plantation culture in other parts of the South and saw no advantage in supporting rich white plantation owners.

I've heard many people talk about how all white people in the South - not just those who owned slaves - benefited from slavery. Economically speaking, however, it seems like poor whites who did not own their own land might have had a hard time finding work, considering they were potentially competing with free labor.

Both statements are true. Poor whites in the south did have difficulty competing with slave labor, but cotton was still the main economic driver in the region; so any work there was to be had was caught up in the fortunes of the large plantations.

Yes, very much so. There's a really good book that goes more into depth about this issue called Free Soil, Free Labor, Free Men by Eric Foner. The author goes over the reasons for the Civil War and explains how slavery & the aristocratic, oligarchic Southerners pretty much devastated Southern economy.