The short answer to your question is that race relations were tenuous at best, particularly between the Afrikaners and blacks/coloreds. One concept that is noteworthy when discussing race relations in pre-apartheid South Africa, is that many different groups held control of various parts of the country at different times, and these different groups held differing views on how to deal with the native populations. It would be useful to define a few terms at the outset to gain a better understanding of how races functioned in South Africa, as well as the different groups at play. Races were defined by ancestry and the physical appearance of a person, which can lead to complications further down the line when pre-apartheid segregation begins. Whites were defined as people with exclusively European ancestry.

As for non-whites, there were different classes that essentially functioned as a sliding scale for how "racially pure/evolved" one was. At the furthest end of the spectrum from the whites were the blacks; people who had exclusively African ancestry. Blacks were deemed the least pure and most unsophisticated race, and were therefore relegated to the worst jobs, had no political representation, and were generally considered expendable, particularly in the eyes of the Afrikaners/Boers. Coloreds were anyone who had mixed European and African ancestry, which can be a particularly muddy classification when no documentation of ancestry can be proven. Some coloreds were so light-skinned they could pass for white, however if it was stated that they were colored on their documents they would not enjoy the benefits that light-skin brought; the same is true in the reverse for extremely dark-skinned coloreds. Coloreds were treated marginally better than blacks, but were still relegated to bad jobs, low pay, and had many of the same limitations imposed on the blacks. One benefit the coloreds enjoyed was very limited political representation, however the efficacy of their political representation is laughable. The other group of non-whites were the Asians/Indians - the terms are generally interchangeable, however the group of people primarily came from India and Bangladesh so Indians is a more applicable term - who were one small step above coloreds, however all of the same racist policies applied to them equally. Indians were imported as slaves during the late 17th and early 18th century by the Afrikaners as field-hands. Indians generally occupied better positions in society - such as merchants or low-level service jobs - than coloreds despite suffering the same impediments as the coloreds.

The arrival of the Dutch in 1652 was initially peaceful. They encountered natives who were mainly comprised of 3 groups differentiated by their means of survival. These groups were the San, Khoikhoi, and the Xhosa, who were hunter-gatherers, pastoral nomads, and settled farmers respectively. In this early period the Dutch were concentrated entirely in the Cape Colony, so territorial conflicts with native populations such as the Xhosa was not an issue. Initially, this settlement was just a small resupply station, but eventually the Dutch government along with the Dutch East India Company granted licenses for farmers to begin to grow their own food to resupply traders on their voyages, beginning conflicts between white colonists and the native black populations. The Dutch colonists began to consolidate their land-holdings, gaining more and more territory, which brought them into violent contact with the Khoikhoi and Xhosa. This period, roughly around 1700-1750, is when racial tensions began to be exacerbated. Tensions within the Dutch colonist community eventually led to the next phase of colonist expansion, and with it, more intensely anti-native sentiment grew. The Afrikaners had relatively good relations with the Khoikhoi and Xhosa who provided the Afrikaners with necessary supplies and cattle, helping sustain the colony through trade, however the Afrikaners viewed the San as pests that needed to be eradicated for preying on their cattle, in the same vein as a rancher killing coyotes. There is a passage from a book called My Traitor’s Heart where the author describes Boers going out hunting San - or as the Boers referred to them, bushmen - for sport but also for the practical purpose of protecting their cattle. In addition to killing all the San they could find, Afrikaners often enslaved them, as well as the other African tribes for use as agricultural laborers.

The consolidation of land-holdings in the hands of a few colonists led to migrations by Boers who set out north and east in search of arable land not controlled by the landed artistocracy of the Dutch colonists, bringing them into contact with more and more native populations, whom the Boers violently opposed in their pursuit of more land. These parties of Boers were raided by native tribes, particularly the militarized societies of the Zulu, who saw the colonists as encroaching on their ancestral lands. The technological advantage granted by guns allowed the Boers to dominate most of these encounters with the natives, causing them to consider themselves superior to the "primitive" natives, although the success of the Boers is not only due to their technological advantage, but internal political cleavages among the Africans, disease, and environmental factors such as native overpopulation and drought. This occurred to a limited extent prior to British occupation in 1795, however the arrival of the British expedited the movement of Afrikaners to the frontiers. From the period 1795-1818, control over the Cape exchanged hands between the British and Dutch as the Napoleanic Wars wreaked havoc on Europe. By 1818 the British were firmly in control and began to settle their own people in the Cape, much to the chagrin of the Afrikaners who saw the British as trying to displace them.

As more and more British settled in the area and began to assert their dominance over the Cape, many Afrikaners abandoned the Cape in favor of the unsettled frontier. One of the main pieces of legislation that enraged the Afrikaners was the passage of Ordinance 50 in 1828 by the British which prohibited holding slaves and mandated that all slaves were to be set free by the middle of the next decade. This led to what is known as the Great Trek which began roughly around 1830 and continued throughout the 1800's. The Afrikaners who left were known as Voortrekkers and they represented the first large scale migrations to the frontiers, as the previous Afrikaners on the frontiers were few in number and had no sizable communities. These Voortrekkers came into contact with the native peoples further in the interior, who often met the Voortrekkers with violent resistance, particularly the Zulus, who had by this time become the dominant power in the southeastern portion of the country where a large amount of Voortrekkers were attempting to settle. The Zulu would raid the Voortrekkers, occasionally dealing a devastating blow to settler parties by exterminating whole communities of people. The Afrikaners would respond in kind by sending commando raiding parties to assist the settlers, resulting in some truly astounding Afrikaner victories such as the Battle of Blood River where roughly 500 Voortrekkers defeated a force of over 10,000 Zulus without losing a single man. These conflicts would inform Afrikaner opinion of the Africans for the next 150 years, making the Boers in particular fiercely racist. The Afrikaners settled into communities that eventually formed their own governments which culminated in the formation of the Boer Republics.

Back in the Cape, the British were treating Africans much better than the Afrikaners, allowing them to hold moderately influential positions such as merchants or low-level civil servants, permitting them to accumulate wealth, and most importantly allowing them to own land. The Boer Republics worried the British and disputes over diamond mines and gold fields discovered in the 1860s and 1870s caused tensions between the Boers and British to come to a head with the two Anglo-Boer wars that occurred 1880-81 and 1899-1902, ending with a British victory and annexation of the Boer Republics. The Boers didn't want to cede their ability to relegate non-whites to the lower strata of society and fought for political representation in the British government, which resulted in the development of a constitution that went into effect in 1910. This constitution gave much more voting rights to the rural farming communities which were mainly occupied by the Boers, which combined with their population advantage allowed them to gain control of the newly formed government.

Thanks to their political majority the Boers were able to push through racist legislation that began to systematize the oppression of non-whites. There were many pieces of legislation but I will only mention the most egregious. The Mines and Works Act of 1911 prohibited Africans from breaking their employment contracts with the mining companies as well as prohibiting them from holding skilled positions in the mines. The Native Lands Act of 1913 restricted Africans from owning land in any desirable part of South Africa, relegating them to just 7% of the total land while they were a majority of the population. The Urban Areas Act of 1923 was the precursor to the pass laws and required Africans to have legitimate business to be allowed in a city and they were not allowed in white designated areas. In addition to all of these pieces of legislation, Africans were not allowed to vote in any province except for the Cape. All of this is decades before the implementation of apartheid in 1948, showing that racism has a deep-rooted history in South African society.

Sources:

A History of South Africa Thompson, Leonard

My Traitor's Heart Malan, Rian