Humans can contain a multitude of contradictions and this is certainly one of them. While there are white Northerners who adhere to something similar to our ideas about racial equality, they are a distinct minority even among people who self-identified as abolitionists...and they themselves are a minority of antislavery thought among whites. So let's unpack that.

To start with defining terms, an abolitionist to whites of the era and usually to historians in he field today, is a person before the Civil War who advocates for the immediate and uncompensated end of enslaving in the United States. This is what actually happened, so it's easy to think back and imagine it as the whole deal but abolitionists were a fringe movement which often declared themselves apart from the ordinary political process and devoted great energy to establishing third parties which accomplished little save to deliver elected office to the party less amenable to their aims. These are people like William Lloyd Garrison, who will go up and burn copies of the Constitution at public rallies and advocate the secession of the North to be quit of slavery.

Most white Americans view them as some mix of cranks and menaces, because at best their ideas are wildly impractical and more often they are destructive to the white American project of self-government. Even whites otherwise quite sympathetic to the abolitionists tend to bristle at being grouped with them and are prone to sometimes shockingly racist denials. Consider this mild example:

I will say then that I am not, nor ever have been in favor of bringing about in any way the social and political equality of the white and black races, [applause]---that I am not nor ever have been in favor of making voters or jurors of negroes, nor of qualifying them to hold office, nor to intermarry with white people; and I will say in addition to this that there is a physical difference between the white and black races which I believe will for ever forbid the two races living together on terms of social and political equality. And inasmuch as they cannot so live, while they do remain together there must be the position of superior and inferior, and I as much as any other man am in favor of having the superior position assigned to the white race.

I must emphasize that an open declaration of white supremacy such as the above, complete with a menu of roles from which Black Americans are permanently excluded, is so common in political speech that it must be understood on the level of invocations of Mom and Apple Pie. Virtually any American of either section would have uttered it entirely without any controversy whatsoever. Its author was then a thoroughly normal, if fairly unsuccessful, politician: Abraham Lincoln.

Yet Lincoln opposes slavery and has done so since his first public speech. He is antislavery, but that means that he wishes to see slavery ended by some gradual, lawful process that may stretch out for a century or more. It would probably be accompanied by a forced population transfer of Black Americans elsewhere in the Western Hemisphere and/or to Liberia, the de facto American colony created for that purpose. This is probably the most broadly respectable antislavery position there is and Lincoln's party advocates not a lot more than banning slavery in the territories, which they imagine will be enough to slowly erode the system to the point it can be easily ended.

This brings us back to your question: with horrific racism like the above fully on the table and uncontroversial, what's the beef with enslaving people? The most trite answer is that one may hate a group of people and still not want some things done to them, just as one might dislike a relative but not want to see them run over by a truck. But it may be a mistake to view Northern racism as inherently milder than the enslaving sort. Whether or not it is depends greatly on what features of racism one views as most salient. Having considered the contrast between enslaver white supremacy and antislavery white supremacy for a good while, my personal take is that they are differently horrific.

Let's take another case study here. During the Mexican War, David Wilmot got up in the House of Representatives and submitted an amendment to an appropriation for money to handle negotiations for the eventual peace. His amendment, the Wilmot Proviso, would have had the effect of completely banning enslaving in any territory taken from Mexico in the peace settlement. It was broadly popular in the North for just that reason and Wilmot stated his purpose plainly:

I have no squeamish sensitiveness upon the subject of slavery, no morbid sympathy for the slave. I plead the cause and rights of white freemen. I would preserve to free white labor a fair country, a rich inheritance, where the sons of toil, of my own race and own color, can live without the disgrace which association with negro slavery brings upon free labor.

Wilmot, like Lincoln, is typical. (Lincoln was also in Congress at the time and voted for the Proviso repeatedly.) Wilmot here tells us that he doesn't care about slavery qua slavery and absolutely,* to the point of throwing shade on the very idea of it*, does not care about what it does to Black Americans. But slavery is a disgrace to white Americans and their free labor system. The knock-on effect of this is that Wilmot understands Black Americans would likely come to the new American Southwest chiefly as enslaved people. By keeping slavery out, he is keeping them out and preserving for the nation's white posterity a lily-white, racially purged west. That West would then as it became states supply votes for national programs to free the American South of the dangerous scourge of Black Americans living there.

There is a tremendous amount of racial hatred in all that, of course. It's also a program that some states of the then-West (west of the Appalachians, in period discourse) had embarked upon on their own. Home to small populations of free Black Americans, they sought to rid themselves of those populations by means of laws which forbade the settlement of any Black American in their territory and mandated the expulsion of those already present. Those jurisdictions included the states of Indiana and Illinois, and the territory of Oregon. Actual enforcement of the expulsion provisions was difficult to manage, but these black laws were very popular among white Americans. When antislavery colonists in Kansas formed a wildcat state government in protest of the illegally-elected one (massive voter fraud; long and horrifying story about how unsubtle and violent such things are when they actually happen) then eagerly voted through black laws of their own and so satisfied some residents who had kept apart from the Free State Party and formed their own Free White State Party (yes, really) that the bigger party was on the up and up. This came over the protests of some within the Free State Party, but they did not win that vote. These people were willing to use violence to keep Kansas free of slavery...and Black Americans.

But I've digressed. Per Wilmot, and many other examples I could name, these dudes have a problem with enslaving. Their problem is that it's bad for white people. That does mean that they dislike the very prospect of seeing a Black American, but it goes deeper than that. They believe that slave labor is fundamentally inefficient compared to free white labor, which further means that it must be propped by by state enactments to give it unfair advantages over free white labor. These enactments are not something an ordinary white man could be expected to prevail against, so he must then sink into poverty. His opportunities for economic advancement will evaporate and he will be left begging for what scraps the enslaver aristocracy will give him.

And the aristocracy is a problem too. White Americans know very well what is done to control black labor: violent terroristic authoritarianism. They do not believe that, in the long run, they can both be desperately indebted to the enslaving class for their prosperity that they can still be immune to that. The men who begin by whipping enslaved people will graduate to whipping free white men. This was not an abstraction; violence against whites in maintenance of slavery was something that happened reasonably often in the enslaving states, up to and including outright murder, and sometimes also outside of them.

Thus enslaving is dangerous to white liberties in the immediate, personal sense. It is also dangerous to them in the general sense of preserving white republicanism. (That is government by the people, for the people, by fairly contested elections and so forth, not the capital-R Republican party platform...though these concerns are in that party's platform at the time.) Northern Whites could look back from the 1850s on an almost uninterrupted history of proslavery dominance on the national stage. Even as the free states grew in number and population, antidemocratic features of American Constitutionalism that are still with us ensured an effectively permanent proslavery governance. These included counting enslaved people toward representation in the House, per-state representation in the Senate, the knock-on effects for the electoral college, and also the special right to recover enslaved people who fled to freedom embodied in the Fugitive Slave Clause. They also included how the South's greater sectional unity over keeping enslaving allowed them to act as a decisive plurality interest even when they lacked a clear majority...as was becoming the case in the last antebellum decade.

How long can the Union bear all of this? And when it breaks, what would happen? Surely a nation divided could not stand against European empires who they imagined sought little more than to destroy it.