It was colonized by both the Dutch and British, but in the grand scheme of things really doesn't have as significant a white population as perceived:
http://www.southafrica.info/about/people/population.htm#.VusDLM4yefQ
About 9% of the country is white. This is probably greater than other colonies because of South Africa's climate (similar to the Mediterranean, allowing for winemaking) and its abundance of resources (gold, diamonds, etc.), giving it greater drawing power than some other African countries
I would just like to point out that until the victory of the black nationalist party ZANU-PF and the emigration of most of the white population, Zimbabwe, then known as Southern Rhodesia, was another African ex-colony with a significant population. Algeria also had a significant white population but that was before independence. You'll notice both of them and South Africa are at the extremes of the continent, and as others have pointed out, that is not coincidence.
Other answers have mentioned disease, cultivation and gold. It's worth expanding on the geography. The southwestern area is a winter rainfall zone in which tried and tested European farming techniques did well and the Niger-Congo techniques worked relatively badly. Consequently the original non-white population was relatively sparse. It also happens to lie around the southern tip of the continent, a natural place for the original Dutch aim of merely having a refreshment station, before colonization started in earnest. This is still the area with the most significant proportion of non-Niger-Congo (i.e. not what is usually seen as black African but including Khoisan/"bushman") peoples. The mineral-rich mining zone known as the Merensky reef lies under the northern plain which is suitable for some now well-known crops (i.e. maize) and farming techniques which whites came to adopt. The other major concentration of whites is here. As you reach the extreme northeast you reach areas with endemic malaria and tsetse fly infestation (somewhat pushed back, at least for a time, by later eradication programmes) deep inland, with relatively few mineral resources. Early attempts at settlement without modern medicine and transport, or to push through to the Portuguese port which is now Maputo to avoid having to use British-controlled Cape Town, resulted in death tolls of 80%+ plus. These areas (and others such as arid areas or the heartlands of better-organised tribes) still contain few whites, and in fact contain large national parks or other government-organized attempts to turn the land to economic account.
Colonial powers did try and establish settler colonies in several parts of Africa. This led to several colonies that had white populations that were between 5% and 10% of the total population at independence.
In 1960, there were more than a million Franch Algerians in Algeria, out of a total population of about 11 million. In South Rhodesia, there were about 225,000 white rhodesians accounting for about 7% of a total population around 3 million people. As others have said, there were also white and mulatto populations in Portuguese Angola and Mozambique that constituted about 5 percent of the total population of those colonies. In Kenya, there were 60,000 british kenyans out of a population of about 8 million, which is less than 1%, which probably stretches the definition of "significant" population.
Each of those countries experienced violent anti-colonial struggles, and the transition to majority rule saw populations of European extraction leave. In some cases like Kenya, the exit was largely peaceful, and a system was organized to facilitate sales of land from emigrating whites to African farmers. A majority of British Kenyans migrated back to the United Kingdom, though some stayed and the population of White Kenyans today hovers around the 60,000 mark out of a population of 45 million.
In other cases like Angola, the Luso-Angolan population was coerced into leaving soon after independence was achieved in 1975. From 1975 to 1979, between 500,000 and 1 million Luso-African Retornados emigrated back to Portugal from Angola and Mozambique.
In still other cases like Algeria or Rhodesia/Zimbabwe, prolonged conflict between majority rule rebels and the French or Rhodesian government left white settlers feeling disenfranchised by the new regimes, and fearful of violence. The vast majority of Pieds-Noirs (white Algerians) returned to France. A majority of the White Rhodesian population ended up in South Africa.
It is true that there was violence in the anti-Apartheid struggle in South Africa, especially in the 1980s until 1994. However, there has not been a large-scale emigration of white South Africans since the transition to majority rule in 1994. There are several factors to point to for this, prominently the reconciliation agenda pursued by Nelson Mandela's administration that acknowledged that white South Africans had lived there for generations and considered the country home.
Just a note to everyone about to comment on this question: comments that simply point out "country X has/had Y% white population" will be removed. Please follow the standards of AskHistorians and ensure you are providing quality answers to the question. It's fine to bring up other countries and their demographics, but please go in-depth as to why that particular demographic profile exists/existed, and how it relates to the demographics of South Africa.