Check out NHGIS, which is maintained by University of Minnesota (https://www.nhgis.org/). They have a very accessible way to download historical US Census data back to the very first census in 1790. While I haven't worked with this vintage of census data, I looked at the 1800 census on NHGIS and see that table NT6 ("Nonwhite Population Except Indians Not Taxed by Slave Status") has "free" and "slave" population counts down to the county level. Hopefully that provides a good starting point!