Hi there -- something that's worth mentioning is that though slavery was absolutely the cause for the southern states to secede, ending slavery did not become a Union war aim until later in the war. There's much more about this in our Civil War FAQ, and I would point you in particular to this post by /u/georgy_k_zhukov, which elaborates on the evolution of Union war aims.

There are several reasons, and you're spot on: it wasn't necessarily about slavery itself per se, though that played a major role, especially when you look at the actions of a radical abolitionist like John Brown, who really helped to increase tensions around the country when he and his men took Harpers Ferry and chopped up several men. It's also fair to say that we as a country were starting to lag behind on abolishing slavery: Britain did away with it in 1833, and France had done so in 1794.

As with a lot of times, the evolution of the Civil War had a lot to do with balance of power, more particularly the number of slave states vs. the number of free states, as this influenced the demarcation of power, particularly within the legislative branch: having more slave states than free gave the slave states a higher sway in the Senate (every state gets 2 Senators, regardless), and due to things like the 3/5s rule (in southern states, slaves counted as 3/5 of a person), this could carry over into Congress as well. Since Congress assigns representatives proportionally to the number of people within a state, this artificially inflated the population of slave states and gave them greater sway here.

The Missouri Compromise is a great example of this: long story short, Maine is admitted as a slave state, Missouri as a free one, to retain the sense of balance between them. In addition to this, it banned slavery in the Louisiana Purchase territory, though the Kansas Nebraska Act of 1854, the next big catalyst on the way to the Civil War, undid this.

Stephen A. Douglas (you may recognize this name if you've read up on Lincoln, from the famous Lincoln-Douglas debates; he was also a successful politician in his own right), reopens the idea of slavery in the territories, as he believes that the decision on slavery is not top-down imposed by the government, but should be decided by the democratic will of the people, the idea of popular sovereignty. This undoes the previously mentioned ban on slavery in the Louisiana Territory-instead, people get to vote for whether they want to live on a slave or free state themselves. While at first the idea is to ignore the inconvenient language in the Missouri Compromise, eventually it leads to a repeal of it entirely, which causes a split between northern and southern Whigs and hastens the death of their party, leading to the creation of the Republicans soon after.

As you might be able to see coming, this leads to a battle of sorts over whether this territory is going to be included in the Union as a slave or a free state. Slaveholders move to the area and bring their slaves with them, while abolitionists and others who frown upon slavery move there as well. This turns them into a powder keg, in essence: Missourians cross into Kansas for the point, basically, of skewing their elections in favor of their pro-slavery agenda, so the issue is heating up rather radically and becoming very contentious by this point.

John Brown, previously mentioned, makes his move around this time period: violence between Free-Soilers and pro-slavery settlers is heating up, and even in the Senate, things are getting violent. Preston Brooks, cousin of South Carolina Senator Andrew Butler, attacks Charles Sumner on the floor of the Senate and beats him with a cane for insinuations about his cousin, feeling it has besmirched his honor. This increases tensions and turned Butler into a martyr of sorts, deepening the divide between North and South. Shortly thereafter, John Brown and his sons chop up five pro-slavery settlers with broadswords, invade Harpers Ferry, an arsenal, and attempt to incite a slave rebellion that does not come. A slave rebellion is effectively the worst fears of the South and pushes us much closer to the brink of war (in my military history class, we were taught this was one of the major catalysts to cause the war.)

The final straw, as I understand, came with the election of Lincoln: he did not win a single southern state, and his election triggered the southern states to break off almost immediately, with the capture of Fort Sumter. Lincoln and the Republicans supported banning slavery in all US territories, very unpopular with the South; debates about the legality of secession have been raging since Jefferson's time (Jefferson and Madison helped open up that can of worms via nullification and states' rights) and even played a part in the tension between Andrew Jackson and his Vice President, John C. Calhoun.

In addition, there is a compelling argument by James McPherson in What They Fought For that soldiers did not necessarily understand the ideological underpinnings of what they were fighting for, based on surviving letters from soldiers. (Ironically, if memory serves, James McPherson was also the name of a rather high-ranking member of Ulysses S. Grant's retinue, mentioned in H.W. Brands biography; found that rather confusing, but that's tangential). Southern letters discuss extensively that they are trying to fight off tyranny or slavery, which is ironic coming from a group of people who are fighting to keep others enslaved, and there is definitely room for interpretation or reading the letters another way.

Anyway, I'm already starting to take up a huge wall of text, so I'll just point out that beyond slavery, another point of contention is that it represented two different lifestyles for the developing country that go back to Jeffersonian and Hamiltonian lines of thought. The North was developing into a more commercial nation, along the lines of Hamilton's vision, and the South retained an agrarian lifestyle. This is part of why the North's victory was, arguably, inevitable: they had the majority of manufacturing ability to create weapons, unlike the South, which was still dependent on having a massive enslaved workforce for their lifestyle to continue.

Sources:

H.W. Brands, The Man Who Saved the Union (Grant biography)

H.W. Brands Andrew Jackson

James McPherson What They Fought For

Allan Reed Millett For the Common Defense

David Herbert Donald Lincoln

Daniel Mark Epstein Lincoln and Whitman

Editor Merrill D. Peterson, The Portable Thomas Jefferson (more ideological; there's not much on the Civil War here, just for his nullification/states' rights remarks)