In order for New World slavery to not be about race, we'd need a lot more counterexamples than Anthony Johnson. There are black slaveholders, but there are no white slaves except for a handful of Slavs that show up in Havana circa 1500. They're remnants of the previous Mediterranean-based slave trade. We would also have to ignore the centuries after Johnson's time when, however slavery was ordered in early Virginia, it became a system of whites owning blacks. To point to extremely rare exceptions to an overwhelming trend is willful blindness at best.

But let's look at Johnson's world. He's not one of the first "twenty Negars" that John Rolfe reports coming to Jamestown in 1619, but he does arrive quite early, in 1621. At this point, records know him as "Antonio a Negro." He arrives in a Chesapeake that is not yet a slave society. Human cargoes arrive in fits and starts, usually small groups. Most of them probably didn't come directly from Africa, but instead had previously been enslaved in the West Indies. Johnson, like most of them, probably had some familiarity with European culture and Christianity. His name suggests prior residence in Spanish colonies.

At the time of Johnson's arrival, not enough slaves exist in Virginia for their presence to demand any kind of thoroughgoing system to manage their lives. Slavery doesn't even exist in law. It's unclear exactly what happened on the ground and day to day, but in at least some cases they are not understood as slaves for life so much as a variety of indentured servant who might someday become free. That doesn't mean life's a party, but it's a hierarchical society with many grades of freedom and unfreedom rather than one oriented around a slave-free or black-white binary.

Johnson was sold to the Bennett family and survived a raid on the colony in 1622. He proved himself a loyal, hardworking sort. Johnson probably had experience winning favor like that from his previous life, since it's one of the ways you got ahead in the Atlantic World. Likewise you tried to outwardly assimilate to the locally dominant culture. It seems to have worked for Johnson, since the Benetts let him operate his own farm, get married (to a woman called Mary), and when he had kids they got baptized. These were all things that could be denied to a slave; baptism could even have freed his children, though I'm not sure precisely when that law was changed.

Johnson and his family get their freedom and Johnson sets out for further assimilation. Antonio becomes Anthony and he takes his hyper-English surname. Anthony, Mary, and their four kids follow the Bennetts to the Eastern Shore in 1640 and set up an independent farm. It goes well enough that in 1651, Johnson earns a 250 acre grant for sponsoring the transport of indentured servants to Virginia. That's a big chunk of land for the time. Two of his sons do quite well on top of that. Johnson's plantation burns in 1653, but he petitions the colony for relief on the grounds of longtime residence and loyal service. That gets him a tax abatement to help in rebuilding.

It appears that the Johnsons, once free, were treated roughly as well as whites. It wouldn't surprise me to find out that they dealt with some casual mistreatment or social isolation, but he and others owned slaves and had rights to property (slaves included) which the courts respected against encroachments by white men. That brings us to one of his slaves, John Casar. (There are various spellings of Casar's surname. Casor is the one I see most often.) Casar was a slave of Johnson's who ran off, probably with the encouragement of some of Johnson's white neighbors. Johnson sued for Casar's return and won it, plus some damages.

There are many reasons the courts might have respected Johnson's right to his slave, aside the simple fact of the law. One is that on the ground there's not a lot of division between the enslavers, the enslaved, and indentured whites. We're not looking at Cotton Kingdom era plantations with a hundred slaves, an overseer, and a staff of drivers. For the most part, when the fields need working the owner is out there with the property and probably any children of age.

The customary rules of labor carry over from contemporary British law, rather than a formal slave code. That grants the employer/owner a lot of power, but does also promise some rights to the workers and those seem to have been respected. (Though there's more mistreatment in Virginia than contemporary England and the slaves would have gotten worse than mere servants.) Those included Sundays and half Saturday off, plus a big break about lunch time. Masters also had to provide for their employees, enslaved or otherwise, and often resorted to the courts for discipline rather than taking it on themselves. You have to go ahead to the 1700s to see enslavers as absolute masters of their domains. The gulf between a slave and a servant, while meaningful, is substantially less than it would later be.

Big picture, a disregard for what we'd call human rights is the order of the day. It's not an endless parade of misery, but most of the ordure descends the incline similarly regardless of one's color. That means it can be horrible, but the doors open to whites to advance are also open to blacks. Johnson's the most famous one who got through them.

All of this changes over the course of the later 1600s, as the number of slaves increases and indentured whites become less reliable (in part because they stop dying so often) and less plentiful. There's a long process by which slavery is consolidated and a gulf opens between slavery and other forms of unfreedom at the lower end of the hierarchy. The line is drawn by color and descent. To be black comes to mean being a slave and whites, by contrast are free(ish). To have an enslaved mother, who is black by definition, then makes one a slave by force of law. The cultural space for free black people shrinks, eventually to the point where Johnson's descendants quit the colony.

Sources

Ira Berlin's Many Thousands Gone (from which I've drawn most heavily)

David Brion Davis' Inhuman Bondage (background with regard to the Mediterranean slave trade)

Edmund Morgan's American Slavery, American Freedom (general background on Virginia)