As usual, there's always room for further answers, but there's an error in your statement that 'only a very small % of white men actually owned slaves'; in the South - where the slaves were - the numbers were very much higher than that (see this very interesting post by /u/jschooltiger from a few years back which explains this in detail).