This is a great question! To answer your question let me outline what the political landscape looked like during the election of 1860, essentially the eve of the Civil War. A glimpse of the parties involved will give us insight into how the American people felt. Keep in mind our ideas of "left" and "right" wing that we use today don't really apply to 1860, so I will avoid using those terms when describing the spectrum. During the 1860 election we have 4 political parties:

• The Republican Party - The candidate is Abraham Lincoln who received almost 40% of the popular vote. The general Republican platform on the slavery issue is that slavery should not be extended into new US territories or future states. Namely, they were against the Kansas-Nebraska Act which allowed those newly created territories to hold a popular vote on whether or not slavery would be allowed within their borders. This was a direct nullification of the earlier Missouri Compromise, which would have banned slavery in both Kansas and Nebraska because they both were above latitude 36°30'. Disregarding the Missouri Compromise meant that slavery could possibly continue into new territories, while the Republican Party wanted to simply contain slavery where it already was. The Republican Party would have been seen at the far side of the spectrum for these 4 parties.Although politicians like Lincoln were actually fairly moderate, the party was popular with die hard abolitionists as well. If we go all the way to the extreme end of this side of the spectrum, we end up with the Radical Republicans, with members like Charles Sumner and Thaddeus Stevens. Radical Republicans held the idea that slavery should be abolished and black emancipation granted. The Radical Republicans often directly insulted Southern planters, and were the group that greatly inflamed Southerners into threats of secession. A great example is when Charles Sumner gave an hours long speech on the senate floor about the horrors of the Kansas-Nebraska Act while simultaneously throwing personal insults at plantation owner Andrew Butler by calling him a pimp for owning slaves. Days later he would pay for his speech by being beaten with a cane by South Carolina senator Preston Brookes
• The Constitutional Union Party - Their candidate was John Bell who received about 13% of the vote. This newly formed party was about as centrists as you could get in the 1860s. They wanted to completely shift their focus away from slavery altogether. The party itself was made up of Southerners who did not want secession, but wanted to keep their slaves. The reason this group is typically seen as moderates though is because they were the only one of the 4 parties to not adapt any official stance on the slavery question.
• The Democratic Party - Their candidate was Stephen A Douglas who received about 30% of the popular vote. At this point in time the Democratic party had fractured into 2 distinctive branches formed mostly by geographical lines. Northern Democrats wanted to preserve the Union, keep slavery where it already was, and allow new territories to vote on whether or not they wanted slavery (Known as Popular Sovereignty). Southern Democrats by contrast wanted to actively take part in the say of slavery in new territories (I'll explain more about them below). The Democratic Party would eventually make a clean break, with the Southern Democrats making their own separate party, and the Northern Democrats being all that was left of the original Democratic Party. On the spectrum, the Democratic Party would be off centered far away from the Republican side.
• The Southern Democratic Party - Their candidate was John C Breckenridge who got 18% of the vote but carried almost every Southern state. This party represents the far side of the spectrum opposite the Republican Party. The slavery question was simple for them: new territories should be allowed to have slaves automatically, or at the very least a slave state be admitted for every free state. On the extreme end of this party would be those labeled as "fire-eaters", Southerners who openly spoke about secession if pro-slavery laws were not passed. This is the group that frequently blamed the Radical Republicans for their threats of secession, believing that the Radical Republicans influenced all political decisions.

So now the other part your question: Was there blame laid on anti-slavery forces for pushing away centrists? The answer is yes. Abolitionists who believed slaves should be freed were seen as fairly extreme, hated in the South, and not very well liked in much of the North. Northerners feared immediate emancipation would greatly disrupt the economy and social structure, and typically wanted to keep slavery contained or look at gradual emancipation. They looked at abolitionists with fear that their extreme views would bring complete social upheaval both because it would upset Southerners and possibly lead to secession, but also because it would mean a huge influx of black people to Northern cities. Likewise, Southerners hated abolitionists because their ideas meant the end of their society, both economically, politically, and socially. To them, abolitionists were the epitome of everything they hated about the North. In the end though, abolitionists and the Radical Republicans ironically had far more influence on Southern Democrats and Fire-Eaters than on fellow Republicans in the North. The Radicals extreme views brought out the worst fears in Southerners, and completely drowned out moderate voices like those of the Democratic Party or the Constitutional Union Party.

Sources:

A great book on the politics right before the Civil War, and where I got most of this information from is: Year of Meteors: Stephen Douglas, Abraham Lincoln, and the Election that Brought on the Civil War by Douglas R. Egerton

For opinions on the North and South's views on abolitionists and the attack on Charles Sumner I mostly used information from the book: The Caning: The Assault That Drove America to Civil War by Stephen Puleo