/u/Krhm is correct that it came down to a case by case basis. Peggy Pascoe argues that "Despite its enormous social power, race was neither a natural essence nor a scientific reality, and pinning it down was often a logical impossibility. This was nowhere more apparent than in the courtroom, where plaintiffs, defendants, lawyers, judges, and juries repeatedly puzzled over how to categorize by race."

Most states outlawed marriages between whites and any person of color, with either a degree of ancestry or blood-quantum, the famous one-drop rule where 1/8 ancestry made the person "colored" (or whatever category). Western states had more categories than Southern ones because their populations were more diverse. Oregon's statute prohibited white marriage to "any negro, Chinese, or any person having more than one-fourth negro or Chinese, or Kanaka blood, or any person having more than one-half Indian blood." Pascoe's research showed that policies evolved from crude visual identification in the 19th century, i.e. you look white to me tests, to mixed standards like the Oregon statute, to 1/8 ancestry as more or less standard.

Source:

Pascoe, Peggy. What Comes Naturally: Miscegenation Law and the Making of Race in America. Oxford: Oxford University Press, 2009.