I mean the Dutch colonized South Africa and didn’t officially cede sovereignty of the cape to England until 1814. But to speak generally to your question: the main reason the Dutch failed to continue colonizing or holding its colony in South Africa was primarily caused by the contentious relationships with England and France. In 1651, the year before the Dutch founded the colony at the Cape of Good Hope, England passed the first of the Navigation Acts, which essentially excluded the Dutch from all direct trade with England or its colonies. This kicked off a period of hostilities that lasted for most of the next ~150 years, with Anglo-Dutch wars starting in 1652, 1665, 1672, and then 1780.

After the first three Anglo-Dutch wars, the most significant outcome was that the Dutch lost New Amsterdam and no longer had a stake in North America. After the end of third war in 1689, Dutch prince William of Orange became King of England, Scotland, and Ireland, ending hostilities between the nations for the next couple generations.

Meanwhile the Dutch were also having to contend with persistent issues with France. The same year the third Anglo-Dutch war started, France invaded and wreaked havoc on the Netherlands. This was an interesting time because the English started as fellow travelers/allies with France, and by the end of the war England had made peace with the Dutch and eventually joined the war against France and speeding up the peace process. From that point it was clear to the rest of Europe that France was the biggest threat and the next few decades were spent dealing with them. Even after William III died and the English throne reverted back to the Stuarts, England and the Dutch republic continued as allies against France, particularly in the War of Spanish Succession (1701-1714).

It wasn’t until a few generations later that Anglo-Dutch relations fell apart again. During their time as allies, England had slowly become the dominant partner and the Dutch lost their control over global trade. By the time the American revolution broke out, the Dutch were ready for a split. They refused to supply Britain with troops, and started selling arms to the rebels in America and naval supplies to France. England declared war and the Dutch got creamed, showing how significantly they had fallen from being a major power. A few years later they were invaded by Napoleon and made a client state with Louis Bonaparte as King of Holland. England, at war against Napoleon, took advantage and captured South Africa in 1795, and after another ~20 years of shifting control during the Napoleonic wars, South Africa was ultimately ceded to England in the Anglo-Dutch treaty of 1814.

So the general tldr is: The Dutch tried, but they were cursed with being a small, easily invade-able country that ended up being at alternating odds with 2 different neighboring superpowers that repeatedly used and devastated the Netherlands to fight their wars (for instance the war of Spanish succession was mostly fought over and in the Spanish Netherlands and the peace treaty was signed in Utrecht). They tried allying with England but that ultimately just weakened their status as a global trade superpower, and they ended up spending most of the colonial era on the ropes, having to dedicate resources and manpower that could have otherwise been used for colonization, to fighting prolonged and often devastating wars at home.