Yes, there was more to it. And technically, the Confederacy didn’t have political parties so there were no “Southern Democrats” during the war (but I will refer to them as such because that’s easier than dividing them into pre-war Southern Democrats and wartime Confederates).

But you also have to remember that the Northern Democratic party was sharply divided during the war, between the War Democrats (who backed Lincoln, the preservation of the Union, and at least limitations on slavery even if they did not support abolition) vs the Peace Democrats ("Copperheads") who wanted a peace treaty with the Confederacy.

At the big picture level, the Democrat Party of that era traced itself to Thomas Jefferson and the small-government/anti-government agrarian path of politics. They were the party of the common farmer or urban artisan (or "mechanic"). You could think of it as the common man/underdog outlook. Or an anti-elitism strand of political theory. That political orientation ran through many Democratic policies; before, during, and after the war.

Take one example: the railroads. Some people (Federalists, Whigs, and then Republicans) supported state governments setting up corporate charters to allow railroad corporations to be formed; to limit the liability of the investors in the railroad, and to either give land away to the railroad corporation or allow the railroad to use eminent domain to take land from farmers to build rail lines. Sometimes state money was directed invested into these railroad companies. No one doubted that this would connect various points and speed trade. Democrats (north and south) were more opposed to railroads because they either opposed these policies which they saw as making the rich become richer, they allowed the investors to use the levers of government to condemn a farmer's land via eminent domain for the railroad, did not believe public money should be used as industrial investments in this manner, or believed that these policies created unfair monopolies (or all of the above). They considered the railroad investors to be a special interest group that the government should not be favoring with handouts and special powers; this was antithetical to the Jeffersonian ideal of independent and informed citizen-farmers operating a democratic government.

In contrast, early in the 19th century, the "cotton capitalists" of the South thought their interests were fully aligned with the New England financier class. The cotton capitalists supported internal improvements, national banking, interstate commerce, etc; because it made getting their cotton to market easier. They saw themselves as capitalist birds of a feather; the rich and "well born" everywhere were first Federalists and then they were Whigs; just like the Northern Industrialists. New England and New York financiers and banks and merchants and shipping companies financed much of the mechanisms of slavery. That's why in 1861, in the few months before the war started, the mayor of New York City wanted the city to secede and join the Confederacy (because of the deep financial ties between the City and the region). Internal improvements - like using the Army Corps of Engineers to make the Mississippi River from St. Louis to New Orleans more navigable. In contrast; a lot of small time farmers in rural Wisconsin or Indiana thought they had more in common, politically, with small time farmers in the South than they did with the industrial and finance-class in New England, New Jersey, Delaware, Philadelphia, etc.

Most Presidents before the Civil War were Democrats, and most of those were from the south (other than Franklin Pierce (New Hampshire) and James Buchanan (Pennsylvania). The Northern Democrats generally agreed with the Southern Democrat sentiment of a smaller and weaker federal government. Pierce did support a transcontinental railroad, and had Jefferson Davis (future Confederate President, also a Democrat) do some surveying on whether that could be run through southern Arizona. Pierce took a middle-road position on the expansion of slavery into Kansas and Missouri; which lead to the horrors of Bleeding Kansas. When Buchanan faced a national banking panic and the failure of tons of bank and businesses, he (1) refused calls for bailouts, (2) blamed speculators, (3) urged lending limits by banks, and (4) took an anti-inflationary position. Again, this was popular among Southern Democrats (there were fewer banks or businesses in the rural South) and popular among the unbanked, agricultural, rural areas of the North (who viewed this as an urban/industrial problem).

During the war, the Northern Democrats picked up seats during the 1862 mid-term election. Part of the platform was that the Federal government in Washington had grown radically, the government had reoriented itself towards industrial capitalism which seemed to only enrich a few businessmen who got lucrative government contracts (while drafting boys off the farm to fight), inflation was going up, taxes were going up, and to top it off, the war wasn't going very well. The North had been told the war would be swift and decisively won; the First Battle of Bull Run in summer of 1861 was supposed to be one huge clash that would decisively end the war. A year later the war was still sort of meandering along (at least in the east and perhaps in Kentucky) and significant troops were being lost to disease in camp rather than a sharp clash akin to a large Napoleonic battle that a lot of people had anticipated. The argument was, why did we expand the Federal government and raise a huge Army and now Lincoln's generals are unwilling to use it and the soldiers are dying of malaria instead.

On the individual level, from the Northern Democrats perspective: if it's 1863 and you are a farmer in rural Ohio or an Irish immigrant in New York City and you don't personally care one way or another on slavery (or you're antagonistic to Blacks, as was common), but someone has come to draft you (or your sons or spouse) into the Army, the major issue of the day now personally effects you. It becomes very easy to become a Peace Democrat and opposed to the war, even if you don't have anything else in common with Jefferson Davis or the Fire Eaters of South Carolina.