As background: At the time of the end of the transatlantic slave trade to the US in 1808, there were fewer than 1 million slaves present in the US. By 1860, there were just shy of 4 million slaves, so that gives an idea of the number of people who were born slaves. A massive number of people were slaves from the day they were born.

Unfortunately there isn't as much primary source material on the childhoods of slaves as we would like. The Slave Narrative Collection indicates that 48% of interviewees described doing work of some kind before the age of 7, though there are accounts of even toddlers working in the fields. While it's a bit afield from the scope of your question, it absolutely bears repeating here that slavery was an atrocity even in childhood. Keep that in mind, as the cultural idea of the slave owner's kids playing with the slave children tends to romanticize slavery and perpetuate the myth of the "good" slave owner.

Slave children did of course engage in play, though any toys they had would have been improvised or, more rarely, made for them by another slave.

A common game mentioned in numerous narratives is “Hide the Switch” a child would hide a simple switch cut from a tree and whoever found it would then chase the others in attempt to whip them. Perhaps this can be viewed as an attempt to deal psychologically with the constant fear of the lash and the authority of the Master.

Another source references common songs sang by slave children during play (slurs obviously present in the originals):

My old mistress promised me,

Before she dies she would set me free.

Now she’s dead and gone to hell

I hope the devil will bum her well.

or:

Run n****r, run.

De patteroll git you!

Run n****r, run.

De patteroll come!

Watch n****r, watch.

De patteroll trick you!

Watch n****r, watch.

He got a big gun!

These examples should show that even play was preoccupied by the psychological (and often physical) torment that weighed on child slaves.

It was common for white children to play with slave children at younger ages, before the white children were school aged (6-8). While many of these children would have been the offspring of poor whites, master's children playing with slave children was common enough, though experiences under slavery were widely varied.

The experience of slave children cannot be generalised, it varied in accordance with a number of factors including time, place and location e.g slaves working on sugar and rice plantations faced greater physical work than those on tobacco plantations.

Additional Reading:

For a broader, and I think necessary, overview of childhood for slave children, the National Archives has a piece titled Childhood Slavery in North America (PDF) The reality faced by most slave children was just as brutal as it was for adults, with punishment beatings, sexual abuse, malnutrition, privation being rife.

There's also A Day In The Life of an Enslaved Child (PDF) which also touches on play for slave children. It's detailed, but as it's meant for an elementary level, is is pretty accessible if you want a historical ELI5.

For more extensive reading:

The National Archives document appears to broadly be a summary of a much larger paper: The Play of Slave Children in the Plantation Communities of the Old South, 1820-1860 (PDF) by David Wiggins, which expands on the master/slave dynamic in more detail.

Documenting the South has an online bibliography of autobiographical material from former slaves, many of which are available online.

I'll leave you with a final quote from Wiggins as a lengthy but good TL;DR:

Much of the leisure time of slave and white children was spent in getting into mischief and helping each other out of difficult situations.... “Me and young master had the good times,” recalled Jack Cauthern of Texas. “He was nigh my age and we’d steal chickens from old Miss and go down in the orchard and barbecue ‘em.”.... Notwithstanding those occasional friendships, a caste system frequently operated within the “play world” of the slave and white children just as it did in the everyday affairs of the plantation community. Older slave children in particular were often forced to assume a subservient position in their game playing. Many of the white children of advanced age were anxious to assume their position as ‘‘superiors.’’