This is an extremely broad and complex question with lots of answers. It's sort of like asking, "Why did Rome fall?" That doesn't mean it's a bad question. It's just hard to answer, especially in a reddit post.
I've always been persuaded by technological arguments. Dutch innovations which allowed sailors to modify their ships with a separate topmast in case of dangerous weather and to track the speed of their ship with the log-and-line still preceded the merchant revolution by nearly a hundred years. ^1 The invention of a Dutch cargo ship, the fluytschip, could reasonably be considered a catalyst for increased trade. ^2 Basically, I think it's reasonable to assume that you need coasts, ships, sailors, a culture that encourages trade, the scientific and technical knowledge to sail across the world, the diplomatic and cultural relations and access routes to be able to take advantage of trading. Countries like Britain and the Netherlands had these things. Without them, colonialism isn't possible or worth the trouble.
I'd hypothesize that a solid banking system would need to be in order as well. International trade is expensive. It requires credit. That kind of credit requires banks. I'd wager a guess that the Netherlands had a sophisticated credit economy during the early colonial period.
^1 Bryan Bunch and Alexander Hellemans, The Timetables of Technology (New York, NY: Simon and Schuster, 1993), 135.
^2 Ian Friel, Maritime History of Britain and Ireland c.400-2001 (London: The British Museum Press, 2003), 168.
Something that should be higher on the page: the Dutch were not an early colonial power. They became a global presence well after the Spanish and Portuguese, and the Dutch maintained a significant world presence well into the 20th century, especially in the populous (and profitable!) East Indies.
So it's completely mistaken to characterize the Dutch as an early colonial empire that fell off early on. They were decidedly 2nd generation, and while not as successful as the English, had some huge enduring successes. Sortof hilarious that we're talking about them crumbling early on when the biggest blow to their empire was Japanese invasions in the 1940s. Let's get our history straight, team.
The Republic was succesful because a number of factors, main ones being:
East Sea Trade: The Republic only briefly supported Denmark against Sweden, most of the time it could freely (well, after paying the toll) import cheap grain from Poland. This grain was used to feed cattle, which is much more valuable per square meter. It is thought this may have been even more important than the spice trade in acquiring wealth.
Wars: While devastating to the southern Netherlands, the Eighty Years' War did allow the Republic to take over Portugese colonies like Formosa (Taiwan). Another source of income was piracy 'privateering', like raids on Spanish treasure fleets. The various wars with England also encouraged investing in fleets and naval technology.
Far East Trade: The Republic controlled the Cape of Good Hope (South Africa), Ceylon (Sri Lanka) and abovementioned Formosa (Taiwan) - as well as other posts along the route to South East Asia. The Republic was able to dominate much of what is now Indonesia, aided in part by geography (relatively narrow islands). It also was the only Western country that was allowed to trade with Japan until 1853. The Republic's pragmatic approach helped, it was interested in money, not in converting foreign heathens.
The Netherlands had traditionally been wealthy and oriented on trade (cities like Kampen were part of the Hanze network). Because they had the money, they could invest in fleets which turned out to also be lucrative, especially because it allowed them to set up the spice trade using conquered Portugese posts as a springboard. And while the Republic wasn't as tolerant as most people believe, its approach to Jews allowed it to draw them in, along with their trade networks and wealth.
As for its downfall. In 1672 the Republic had about 1,5-2 million people. England had a population of 4 million, France was at 20. Outnumbering their enemy 12-16 to one, not even counting Cologne and Münster, they decided to wipe the Republic off the map.
At the time the Republic was going through an economic, political and military crisis. The civilian government had done away with Stadholders, but the Oranges (in the person of Willem III) still wielded influence and power. While the fleet was by necessity well-funded, Johan de Witt (who was basically the prime minister) didn't trust the army and underfunded it.
Now the combined Anglo-French fleet threatened the north and west, while a vast French army (100.000) marched up from the south, supported from the east by the bishops from Cologne and Münster. The Anglo-Franco-German coalition occupied everything aside from Holland/Zeeland/Friesland/Utrecht.
Eventually, through alliances, superior admirality and vast investments in the army, the Republic won. But it was a Pyrrhic victory. When the United Kingdom of the Netherlands was formed in 1815 it was finally able to use the wealth of what is now Belgium to pay off the debts acquired during the 'Franco-Dutch' War.
And the Rampjaar was only the lead-up. In 1688, Willem III (who was made Stadholder after Johan de Witt was lynched and partially eaten) conquered England with a fleet twice the size of the Spanish Armada. While this eventually allowed him to form a force capable of stopping the Sun King in the War of Spanish Succession, it was another Pyrrhic victory for the Republic. King Willem III took seat in occupied London, and a sizeable amount of the banking and trade sector followed.
The Republic never recovered from the loss of income and the burden of debt, but it seems unlikely it could compete with its more populous creation, the UK, anyway. It had become a junior partner, especially after the death of Willem III. The War of Austrian Succession made it painfully obvious that the Republic wasn't a major power anymore. The crippling blow ironically came from the UK, in the Fourth Anglo-Dutch War.
At the turn of the century, it was casually annexed by Napoleon. In the 19th century, the newly constituted UKNL suffered from poor leadership. Belgium broke off in 1830, and even they were more powerful and wealthier, able to take part in the Scramble for Africa for instance.
TL;DR: Succesful because of money, didn't last because of money.
User /u/Wojiz made great points, but another event that put the Dutch on the international trade map was their success in breaking the monopoly of the Hanseatic league in 1441 and gaining maritime access to the Baltic Sea and all the other Hansa seaports. As a supra-national corporation, the Hansa transcended the turbulent political mess of medieval Europe and allowed Dutch trade to flourish. This was the kickstarter for Dutch technology, ships, banking and mercantilism to break out to lands further and wider.
One thing people forget about is the Dutch influence in North America was due to King James banishing settlements the area between Plymouth Colony and Virginia Colony. Very early on he declared that no settlement would be allowed within 100 miles of each colony. The northern border of Virginia was what we now call the northern border of Maryland and the southern most point of Conneticut was considered the southern border of Plymouth. This area between the two became what American's call the Mid Atlantic and was much different that the two other areas.
As there was no presence of British settlements in this no mans land the Dutch and other countries took advantage and developed a trading empire with the Indians. The trading posts on the Delaware and Hudson Rivers sprung up. But these at first were very limited in population. It was not until later that there were efforts to create settlements with larger tracts of land being given to people. These became successes like Manhattan, Albany, and towns on Long Island.
Eventually the English had enough as the colonies were trying to trade with the Dutch and avoid paying taxes on goods imposed by the Navigation Acts of the 17th century. The British and Dutch fought 3 small wars and then at the end the Dutch left North America.