I think the first folly is to think of South Africa, the modern nation-state, as a whole entity, when in reality, the modern territory of South Africa were under a variety of different ethnic groups, and even today it is extremely ethnically diverse.

The largest ethnic groups in South Africa are Zulu and Xhosa people, and historically, these have been in the Kwa-Zulu Natal region, and much of Northeast South Africa for close to a millenium, having arrived in the Bantu migrations long before European arrival.

However, these groups belong in eastern and northeastern South Africa, areas where European colonists did not arrive in any meaningful numbers until the Boertrek in the 19th century. In the Cape region, where the Dutch arrived in the 1600'es, the image is different. The people residing around the Cape were the Khoikhoi, whom the Europeans derisively referred to as the Hottentots. These were Khoisan pastoral people, and their population were comparatively small. These Khoikhoi were the ones leading engagement with Europeans in the Cape Region for two centuries, and they are the ones who suffered from wars and disease, and displacement from their lands as the Dutch colonizers expanded. The khoikhoi are today an extreme minority group, to the extent that they have almost disappeared - a total of 100.000 still remain, divided across Namibia, SOuth Africa and Botswana, making them a very small percentage of SOuth Africa's population. Today, the vast majority of blacks in the Cape region and Western South Africa are from the eastern part of the country, having arrived after European colonization.

In short, Western South Africa was not empty of people when the Dutch arrived, but it is today largely empty of the people who lived there pre-European arrival. Hence, this has given rise to the somewhat misleading image that the White SOuth Africans are the oldest ethnic group in the country, having been in the Western region (but not the eastern) longer than the Zulu and Xhosa.