You might be interested in The Making of a Confederate: Walter Lenoir's Civil War. It's a biography about a Southerner who opposed secession and struggled with the morality of his own slave ownership. He was planning on moving to the North to finally rid himself of the burden. But when the war broke out, he saw the North's aggression as egregious, joined the Confederate Army and was seriously wounded in 1862. He lived however and devoted his life to the lost cause and always saw the war as a defense of his home and not one over slavery.

While I'm not sure how many people were like Lenoir, people of his opinion did exist.

Check out the book here

I have a related question - I know I've read about Seminoles (some of whom were "maroons" or cimarrónes, escaped slaves) who fought on the side of the Confederacy... the rationale there being that they'd just fought a series of wars against the Union and anyone who was shooting at those soldiers in blue was A-OK in their book.

Don't know how accurate that is, though, so I'm curious what the consensus is on Confederate Seminoles. A real movement? One or two guys? A myth?

This isn't entirely on-topic because he was a Northerner, but Lysander Spooner was an interesting example of anti-Union and anti-slavery sentiment. Spooner oftentimes get played up by libertarians and Civil War revisionists as an example of a conscientious states-rights proponent (how accurate this is, I'm not really sure, so please correct me if I'm wrong).

Spooner was both a harsh critic of the Union/Lincoln/the War and of slavery. He famously argued that slavery was unconstitutional, but also felt that secession was a vital function of self-determination, and that the invasion of the Confederate States of America was a form of aggression. He later published a series of essays called "No Treason", in which he claimed that the social contract that justified the United States Federal Government (as a compact between consenting agents) had been irreparably violated by the Civil War, and the Union no longer maintained any legitimacy.

Spooner's life gets appropriated by both "left" and "right" libertarians for this reason (he was an anarchist and a critic of the Federal Government and took a pretty consistent stance against slavery), so he has some modern fame amongst heterodox historians. Now, that said, he was from Massachusetts and he was not well-received in the South, so this isn't completely en-point, but he seems like the sort of person you're wondering about.

Edit: The reason why I thought he was particularly relevant (despite being a Northerner) is because he actually had a principled reason for opposing the "War of Northern Aggression" while also opposing slavery. Many of the examples of Northern opponents of the war were either Southern sympathizers (who supported slavery) or were against the War because of its costs (too many casualties, institution of the draft, etc. etc.). Spooner was against the war for a reason consistent with his philosophy (his reason for opposing slavery - the right to self-determination - was the same motivation for his opposition to the war).

I think is also important to make a distinction between slavery and racism, some in the South may have opposed slavery as an institution, Robert E. Lee has been mentioned already as an example, these people thought the institution was wrong and immoral but the vast majority of the country (both South and North) was overwhelmingly racist. Slavery may have been opposed as an institution but there was little commitment to actually let the black population be equal in all levels as history has shown.