It's important to remember that one didn't have to directly own a slave to come from a slave holding background. Most men who joined the army were relatively young and had not had time to accumulate wealth and, consequently, slaves. The figures given below are from Joseph Glatthaar's General Lee's Army.

Slightly more than 10% of the men who volunteered for the Confederate Army in 1861 owned slaves. In addition to that, slightly more than 25% of the men going into the army lived with parents who were slave owners. This gives a combined total of 36% either slave owners or from slave owning nuclear families, as compared to 24.9% for the Confederate population as a whole, making the Confederate soldier 42% more likely to have a direct connection to slavery than the average Confederate citizen. If you add in those men who lived in households headed by a non-blood relative, the figure rises to just under half of all Confederate soldiers having a direct, pre-war link to slavery.

Officers were especially likely to own slaves; more than half of all Confederate officers in 1861 were themselves slave owners.