Yeah, so this question indeed seems a puzzling one. Using the data of the 1860 census, you find out that only 5 to 8% of Southerners owned slaves. Using only the owner is not the adequate metric, since, for example, only the male patriarch would count as an owner despite the fact that all of his family would benefit from the slaves. Yet, even if we measure by how many families held slaves, the percentage seems still somewhat low - around 30% of families, going as high as almost 50% in Mississippi and South Carolina. Even among those families, many were poor yeomen who had to till the land alongside their slaves. Why the other 70% of the South would fight for such a privileged minority is not clear to many, but the answer is simple enough: the defense of a White Supremacist South.

There is no sugarcoating it. Southerners, simply put, fought for slavery and White Supremacy. You may wonder why they would do so, even when they did not own any slaves and the fall of the slavocracy would probably benefit them. One of the great failures of the Republicans before the Civil War and during its opening stages is that they too believed that poor Southerners without slaves would not fight for the Confederacy, but would instead uphold the Union. Indeed, people in both North and South believed that non-slaveholding whites were the key demographic through which slavery would be exterminated. “There is no denying that there is a large emancipating interest in Virginia and Kentucky and Maryland and Missouri", declared the Fire Eater William L. Yancey. The idea that non-slaveholders were the victims of deceit by the slavocrats, and that they would soon return to their senses and rally round the Union persisted for many months, but soon enough Union military and civil leaders learned that the Confederacy actually enjoyed enormous popular support.

It's true that areas with fewer slaves and wealth tended to elect conditional unionists to the conventions that passed the ordinances of secession. But the key word is conditional. For many of them, the maintenance of the Union hinged on whether the incoming Lincoln administration would do any "overt-act" that justified secession. Though they were able to stave off secession in the Upper South in the immediate aftermath of the election, after Fort Sumter the great majority of conditional unionists rallied to the Confederate standard instead. The mountainous regions of Upper Alabama, Eastern Tennessee, Western North Carolina and West Virginia, where most of these unionists came, would remain centers of Wartime Unionism that bitterly resented the Confederacy and resisted its policies, such as the draft and taxation. But non-slaveholders of other areas did fight gallantly for the CSA. Again, why?

As Governor Brown of Georgia said, it was because poor yeomen believed that slavery "is the poor man's best Government." Brown continued by declaring that "Among us the poor white laborer . . . does not belong to the menial class. The negro is in no sense his equal. . . . He belongs to the only true aristocracy, the race of white men." Indeed, for White Southerners the destruction of slavery would not affect only slaveholders, but would inevitably result in dreaded equality for Negroes. In vain did Republicans insist that they would not interfere with slavery in the states, but just limit it so that it would disappear gradually and painlessly. By 1860, Southerners were convinced that the victory of any Republican, no matter how moderate, would result in immediate slave emancipation and the destruction of White Supremacy. Southern leaders unabashedly appealed to racism as a justification.

From McPherson, Battle Cry of Freedom:

The election of Lincoln, declared an Alabama newspaper, "shows that the North [intends] to free the negroes and force amalgamation between them and the children of the poor men of the South." “Do you love your mother, your wife, your sister, your daughter?" a Georgia secessionist asked non-slaveholders. If Georgia remained in a Union "ruled by Lincoln and his crew… in ten years or less our children will be the slaves of negroes." "If you are tame enough to submit," declaimed South Carolina's Baptist clergyman James Furman, "Abolition preachers will be at hand to consummate the marriage of your daughters to black husbands." No! No! came an answering shout from Alabama. "Submit to have our wives and daughters choose between death and gratifying the hellish lust of the negro!! . . . Better ten thousand deaths than submission to Black Republicanism.”​

To defend their wives and daughters, presumably, yeoman whites therefore joined planters in "rallying to the standard of Liberty and Equality for white men" against "our Abolition enemies who are pledged to prostrate the white freemen of the South down to equality with negroes." Most southern whites could agree that "democratic liberty exists solely because we have black slaves" whose presence "promotes equality among the free. " Hence "freedom is not possible without slavery. "

Southerners thus fought to keep slavery, even if they did not have any slaves themselves, because slavery secured White Supremacy, a system that benefitted White men and kept them as social superiors. Because no matter how miserable, how poor, how uneducated they were, White Southerners would always be better than the Negroes, always have someone they could down on. Even when they talked of defending their homes from Yankee invaders, or protecting States Rights against an overreaching Federal government, it was understood that the reason why they needed to defend their homes was because otherwise slave emancipation and Black equality would follow, and that the States Rights they defended were about slavery and the condition of Blacks in the South.

I'm not exaggerating, nor being vitriolic. Southerners candidly admitted that Black liberty was a threat to their way of life, and that Black equality terrified them. Racist rhetoric played upon this fact, and Southern leaders called on their people to join the army to protect against the Black Republicans that sought to free the Negroes. They admitted to the fact that they fought for slavery. A Kentucky private, showing the fact that liberty was only for whites, declared that they were "fighting for our liberty, against tyrants of the North . . . who are determined to destroy slavery." Others were shier, and used "southern institutions" as a code word for slavery. Others were bolder, and vowed "to fight forever, rather than submit to freeing the Negroes among us . . . We are fighting for rights and property bequeathed to us by our ancestors."

These soldiers were slaveholders, but the rhetoric of the poor was similar. "I never want to see the day when a negro is put on an equality with a white person", said a Louisiana soldier, while a comrade from North Carolina said that he fought to show the Yankees "that a white man is better than a nigger." When Confederate morale sagged and it seemed like a Peace candidate would win North Carolina's governorship, the war candidate rallied his supporters by telling them that "Instead of getting your sons back to the plow and fireside, they would be drafted . . . to fight alongside of [Lincoln's] negro troops in exterminating the white men, women, and children of the South." An Arkansas man declared that he had to fight because otherwise his "sister, wife, and mother are to be given up to the embraces of their present male servitors." A Georgia soldier summarized these fears and prejudices when he said that if the Union won "we are irrevocably lost and not only will the negroes be free but . . . we will all be on a common level."

It is true that some Confederate were impelled by other motives. Some because they saw the Yankees as invaders who would take their lands, property and lives; others, out of loyalty to their states (Robert E. Lee, for example). Loyalty to a commander or to their bataillon was also common, and in the opening stages of the war many Dixie boys enlisted due to social pressure or because they wanted adventure and thought of war as short and glorious. But no Southerner ever doubted that he had to fight to prevent Negro equality, and that an independent Confederacy would be built on slavery for the benefit of White men. There was a lot of resentment over the idea that they were fighting for the benefit of rich men, and soldiers did not respond well to pro-slavery measures such as the fact that overseers were exempted from the draft. Sometimes, the abstract goal of White Supremacy was superseded by the concrete wish of returning home or escaping danger. As in other armies, there were deserters, draft dodgers, and men who skedaddled and never saw combat. But for those Confederates who fought till the bitter end, there was an understanding that they fought for an independent Confederacy, where slavery would be perpetuated and White Supremacy would be the law of the land.

Sources:

James McPherson's Battle Cry of Freedom and For Cause and Comrades. The later is probably the best book if you want to know the reasons that motivated soldiers in the struggle. These are the two main sources, but Oakes' Freedom National, Foner's Free Soil, Free Labor, Free Men, and Porter, Impending Crisis, also form the basis of my knowledge.

This older answer is one you might find of interest https://www.reddit.com/r/AskHistorians/comments/67fvaf/was_the_average_confederate_soldier_a_strong/dgq8tn2/