The idea that black South Africans are "not indigenous" and therefore are no less "invaders" than colonists is an old settler myth, used to undergird apartheid justifications and colonial dispossession via the uprooting of native title in a general, race-based way. It remains politically popular among some groups of people, just as its absolute opposite does (that Bantu-speakers' presence anywhere in the subcontinent invalidates all white South Africans' claims), but it is just as simplistic and incorrect. In fact, South African scientists and ethnographers long knew that mixed agriculturalists using iron--and whose material culture matched people north of the Limpopo--were present before 1500, all the way down to the Kei. During the apartheid era, they weren't generally allowed to say that these were Bantu-speakers, or to link them with any living groups of people. However, there were various S-group Bantu language speakers everywhere east of the 400mm annual rainfall line (about 26 degrees E longitude), slowly moving west and south, starting no later than 1400 or 1500 years ago.

As for indigeneity, people did in fact identify differently in the past--totemic and patronymic naming changed with allegiances--but that is the equivalent of saying that white South Africans all arrived after 1795 because the Dutch East India Company "vanished." The families still exist; Xhosa, for example, was a real person who lived near the Mbashe or Qhorha rivers around 1700. Generally, the passage of early Bantu-speaking communities to the south of the Limpopo is pegged somewhere around the sixth or seventh centuries, as per Peter Mitchell (Archaeology of Southern Africa, 2002) and more recent works. Certainly, large settlements of hundreds or thousands of family compounds existed by the tenth century, including the Toutswe complex and of course Mapungubwe in the north; more recent is the massive complex of settlements collectively known as "Bokoni" stretched across Mpumalanga, which emerged between the fifteenth and seventeenth centuries, but we have evidence of homesteading all over the region that goes back up to a thousand years and does not fit the pattern we ascribe to Khoisan people. (That said, Khoisan tended to merge into Bantu-speaking societies over time, so it's hard to know the exact dynamic.)

It's important to point out, despite its obviousness, that "South Africa" was not one place to arrive. It's a very big country, with a wide range of climates and topographies, and it is true that the southwestern Cape and most of the area west of the 400mm isohyet were not the homes of Bantu-speaking states and communities in the era before 1800.