Several slave States remained within the Union during the Civil War. This includes Missouri, Kentucky, Maryland, and Delaware.

In the 1860 census the slave populations of those States were 115,000, 225,000, 87,000, and 1,800 respectively. Additionally Washington DC had 3,000 slaves at the beginning of the war. All of this was a small fraction of the 3,950,000 slaves that lived in America. All of these states had a much larger population (of whites plus slaves). 1,182,000, 1,155,000, 687,000, and 112,000. That is under 10%. Contrast that to most other slave states which had a ratio of about 1/3 slaves.

These border states were very divided on the topic of slavery. Many men in these states left the state and joined the confederate army. On the flip side, many men in Tennessee and Virginia broke away from their state government and huge regions remained loyal to the Union. West Virginia became a border state in this manner.

In regards to prejudice, it was relatively low in places like Boston but quite high in regions like Oregon and Illinois where laws existed that prohibited free blacks from moving into those states. Of course that prejudice is still way lower than the prejudice against blacks under the institution of slavery.