I'll try to answer this, even though this is far from my area of expertise. It can be boiled down to a series of factors :
The first factor is climatic
a. The Dutch Republic benefited greatly from the herring catch in the seventeenth century (see Vermeer's Hat by Timothy Brook). The global cooling that took place in Europe iced in established fisheries (in Norway for ex.), and pushed fish stocks away from the North Sea towards the Baltic sea, which was then controlled by Dutch fishermen. This sudden windfall gave the Dutch the capital to finance other ventures.
b. The succession of soggy summers and harsh winters which caracterized the seventeenth century greatly impoverished certain geographical areas with low agricultural yields (see Global Crisis - War, Climate Change and Catastrophe in the Seventeeth Century by Geoffrey Parker). This led to constant grain shortages and civil strifes in countries such as France and England, whereas the Dutch Republic was relatively less impacted by this. Pierre Goubert, in his famous book on French demographic history (Louis XIV et vingt millions de Français), starts off his first chapter by noting that agricultural yields were 30% higher on average in the Dutch Republic than in France.
c. The end of this global cooling phase will gradually diminish these two advantages of the Dutch Republic in the eighteenth century. In conjunction, the agricultural revolution of the 18th century will allow neighbouring countries of the Dutch to greatly increase their population (France will go from 20 million to 28 millions inhabitants during that century, providing a great advantage come the Revolutionary Wars)
The second factor is diplomatic/military
a. The Dutch had benefited in the first half of the seventeenth century from a series of fortunate events (for them!) such as the Siege of Antwerp in 1585 and the Thirty Years' War (1618-1648). The former sent a vast number of entrepreneurs from Antwerp to Amsterdam, which became the new commercial capital of Northern Europe. The latter destroyed for two generations the economy and population of many German states, preventing any serious competition from any Southern neighbours.
b. Successful attacks by England, and than France, will diminish the Dutch Republic's standing on the international scene. The English will strike first in 1652, which will lead to a series of wars between the Dutch and the English and tilt maritime domination towards the latter. The French will strike in 1672, ending several decades of cooperation between the two countries, and starting the rampjaar, or "disaster year". This conflict will end with the Treaties of Nijmegen in 1678, with France gaining a strong foothold in the former territories of the Spanish Netherlands.
The third factor is economic/geographic
a. The Dutch Republic is a country with a relatively small population compared to its neighbours, and is relatively low on natural ressources. Whereas France, England and modern-day Belgium can count on lands with ressources such as coal, iron ore, forest (wood for industries and boatbuilding), the Dutch Republic had a lower amount of lands and ressources to build from. Skipping ahead, this will prove to be a capital point during the Industrial Revolution, with England and modern-day Belgium benefiting more from their ressources and giving way to the first industrialised cities.
b. A consequence of this is that the Dutch Republic couldn't sustain its exceptional success in the seventeen century in the following century because of the limitations described in 3)a. One notable example is the attempt by the Dutch to capture part of Brazil in 1624 (unsuccesfully) and then again in 1630 (sucessfully). Even though they conquered almost half of Brazil's settled European area at the time, they were only able to hold on to these lands for roughly 30 years before having to cede them back to the Portuguese. As a country, they had strecthed themselves thin and therefore suffered a few setbacks.
In conclusion, in can be argued that the Dutch Republic didn't decline so much as stagnate in the 18th century. Sure, in relative terms, they went down a few notches on the global pecking order. But that didn't prevent them from maintaining a high urbanization and literacy rate, developing a thriving economy, and maintaining strong ties with its main colony, Indonesia.