I think the simple answer is that racial hierarchy was natural, at the time, and even if there were arguments that the South got a little carried away with their slave codes their fundamental assumptions about 'If it aint white, it aint right' were not necessarily offensive or incompatible with integration into a cohesive American milieu.

What was incompatible was a separate economic system (i.e.: chattel slavery). When Senator John J. Crittenden proposed to resolve the secession crisis in 1861 by reinstating the 36 30' line between slavery and freedom in all the territories "now held, or hereafter acquired," Abraham Lincoln and the Republicans rejected the proposal on the ground that it "would amount to a perpetual covenant of war against every people, tribe and State owning a foot of land between here and Tierra del Fuego.” Cong. Globe, 36 Cong, 2 Sess., 651.

I think it is an interesting comment because the debate captures something about Northern attitudes towards slavery as it was expressed in the South. Slavery was seen as a proselytizing economic system in violent competition to 'free labor.' Part of this was Bleeding Kansas, part of this was the filibusters (e.g., William Walker, John Quitman and the Knights of the Golden Circle) and part of this was a cold calculus that a free system and a slave system could not easily exist side by side in any state (whether or not that is true).

As hard as it may be for us to understand today, slavery was an incredibly useful and profitable economic system. Racism was a useful justification, and perhaps sincere, but the minds of both the North and South slavery was a separate 'issue' if intimately bound up with racial identity. Slavery predated racism and, in terms of modern sex trafficking, will outlast it.

So if slavery and free labor cannot exist side by side, while slavery (at least by the '60s) was considered a dangerous, expansive influence that would eat up all of Latin America if given half a chance, slavery must end. But none of those reasons readily extend themselves to racial equality over all. Without chattel slavery, there is no impetus for the South to expand. Without the need to expand, the South can do whatever it wants. And, to many in the North, having second-class citizens was not nearly as indigestible as the concept is now today.

Why did slavery 'need' to expand is a different question, but that was the consensus in many circles both North and South. That was what the Constitution envisioned, a slow termination of slavery by limiting its geographic scope through eliminating its international sources and, eventually, by slave owners manumitting their slaves as they moved on to the next life (e.g., Thomas Jefferson).

And that vision continued throughout the lead up to the Civil War. To take one example, when William Walker and other filibusters attempted to invade South America the Northern reaction was that tropical states could only be slave states. To integrate a tropical state into the United States, merely because the United States allowed (not required, merely allowed) slavery the new state would either become a slave state or be a false free state hiding deep Southern sympathies. Slavery was just that insidious and that naturally expansionist; “they deemed the expansion of the domain of slavery an inevitable result of its success.” Robert May makes a similar argument when he evaluates this conodrum in the context Stephen Douglas’ candidacy, and the difficult position he was put in as he tried to navigate the Charleston Democratic Convention’s “tropical agenda.”

Not that it's limited to Douglas, but every Presidential Convention from Ole Tippecanoe onward is dealing with the fight of two economic systems that were seen as fundamentally at odds. But, ultimately, these are economic systems. A 'mere' social hierarchy system isn't very well considered. Even if it is based on racial inequality, especially if it's based on racial inequality.

Why isn't social hierarchy, inequality and so on, not considered? Another question, I suppose.