Depends where you were. For example Oregon had it in their state constitution to prohibit new immigration of African Americans into the state, as well as making illegal their ownership of real estate and entering into contracts.  They were also denied the right to sue in court.  This clause, Article 1 Section 35, was subject to popular vote, as was the adoption of a ban on slavery and the entire constitution.   The exclusion clause received more popular votes than the approval of the constitution or the ban on slavery.  Although enabling legislation was never passed and the clause was voided by the14th and 15th Amendments passed after the Civil War, the ban remained a part of Oregon’s constitution until it was finally repealed in 1927. - See more at.

In Boston they had the immigration restriction league which was not a fan of the Irish, Jewish, Italian(basically any non Protestant whites) people coming into the states. Usually with accompanying pseudoscience about those people groups being inferior with birth defects that predisposed them to criminality.

Then there was nation wide court rulings like [U.S vs Wong Kim Ark] (https://www.law.cornell.edu/supremecourt/text/169/649) where the courts had to rule children born in the United States from parents of Chinese dissent were citizens of the United States(up to that point they would have been considered subjects of the emperor of China).

So long story short Americas attitude towards minorities is usually "their different?! hissss"