A quick (probably wrong) google shows only about 5% of the population in the South owned slaves. Why would an average confederate person risk their life and go to war for something that only affected/benefited a small elite few?
This older answer of mine should be of interest.