Abraham Lincoln actually addressed this very question in his first Inaugural Address, delivered in March 1861.

To put this date in context: Lincoln was elected in November 1860. Before his inauguration in March, seven states had already seceded from the Union (South Carolina, Mississippi, Florida, Alabama, Georgia, Louisiana, and Texas). South Carolina had already demanded that the United States surrender Fort Sumter, and was refusing to allow the United States to send supplies to the soldiers stationed there.

Why couldn't the North just let the Southern states go? Lincoln argued both that the Southern states had no right to leave and that allowing them to leave would destroy the United States.

On the first point: Lincoln noted that there is nothing in the Constitution that allows for a state to leave the Union. He also compared the Constitution to a contract among the states: "One party to a contract may violate it -- break it, so to speak; but does it not require all to lawfully rescind it?"

More concretely, Lincoln pointed out that the logic of secession would lead to national destruction. Lincoln said that there had always been many political disputes, and that in the American system, there must be compromises. One does not always get what one wants: so, for example, the South was angry that northern states did not always return fugitive slaves, and northerners were upset that laws against the international slave trade were not always enforced. When there is a disagreement, one side or the other (whether the majority or the minority) must be willing to let things go and live with imperfect results, or the government cannot hold together. In his words:

"If a minority in such a case will secede rather than acquiesce, they make a precedent which in turn will divide and ruin them; for a minority of their own will secede from them whenever a majority refuses to be controlled by such minority. For instance, why may not any portion of a new confederacy a year or two hence arbitrarily secede again, precisely as portions of the present Union now claim to secede from it?"

Lincoln saw that the logic of secession would destroy the Confederacy. If the Confederate states agreed that the "right" response to an unpopular government or an unwelcome government decision is to secede, they would just keep seceding from each other.

But the same logic would apply to the Union. If the North just let the South go, it would establish the precedent that angry states can leave. States get angry in American politics all the time, so this would be an exceptionally dangerous and damaging precedent.

Imagine that the South was permitted to leave in 1860-1861. How long would it be before California, or the Midwest, or the Great Lakes states decided that they also had some great conflict with the rest of the US and needed to leave? If secession were permitted, it would prove to be an unstoppable force. Lincoln said that his oath to "preserve, protect, and defend" the Constitution would not allow him to watch that happen.

Source: Lincoln's First Inaugural Address, at https://avalon.law.yale.edu/19th_century/lincoln1.asp

The aptly-named /u/secessionisillegal provides some important detail on why "peaceable secession" was unlikely in 1861 here.

Among other issues, they point out:

Division of U.S. property within the Southern states, division of U.S. property out West, the division of national debts, treaties over the recovery of escaped slaves, treaties over control of the Mississippi River that interior Northern states depended on for commerce, treaties over international trade and tariffs, and so on and so on...These were all issues that a "peaceable separation" couldn't address any better than a compromise keeping the South part of the United States could.

What happens to all that federal property that no individual State had paid for, like federal courts, post offices, and most of all military installations (forts, armories, etc)? What about ships that might have been built in New England but were now docked in a Southern port? These were thorny issues to say the least, and though Confederates claimed they were willing to negotiate a settlement for federal property, they had gone ahead and seized it preemptively [1].

A larger issue is that one of the chief political divisions of the 1850s had to do with the fate of new territory in the West, in the lands of the Mexican Cession, and increasingly in areas like Kansas and Nebraska. Southern slaveholders and Northern "Free Soil" advocates had very different ideas for what the future development of those regions would look like. How would that territory be divided? If the South was truly leaving the Union and the Union still existed, then the Confederacy had no claim to those territories, but that ran counter to most of the goals of Southern expansionists.

Indeed, Southern planters, especially in the heavy cotton regions, tended to favor the most aggressive expansionist policies of any group in the US at the time. They had been most in favor of the Mexican War, and they were continually advocating for the annexation of Cuba, and potentially other parts of Mexico and Central America. Senator Albert Gallatin Brown of Mississippi may have been among the most outspoken of his contemporaries, but his sentiments were not exactly rare among the Southern elite, especially his fellow "Fire Eaters":

I want Cuba, and I know that sooner or later we must have it. If the worm-eaten throne of Spain is willing to give it for a fair equivalent, well—if not, we must take it. I want Tamaulipas, Potosi, and one or two other Mexican Stats; and I want them all for the same reason—for the planting and spreading of slavery.

And a footing in Central America will powerfully aid us in acquiring those other states. It will render them less valuable to the other powers of the earth, and thereby diminish competition with us. Yes, I want these countries for the spread of slavery. I would spread the blessings of slavery, like the religion of our Divine Master, to the uttermost ends of the earth, and rebellious and wicked as the Yankees have been, I would even extend it to them.

Thus when Lincoln took office, he and his fellow Unionists were not faced with an isolationist Confederacy that just wanted to be left alone, but rather with an aggressive, expansionist power that was actively seeking new territory and likely would have drawn the United States into conflict sooner or later anyway.

[1] See James McPherson, Battle Cry of Freedom, especially "The Counterrevolution of 1861" chapter.