The Völkerwanderung affected the Frisians too, and it is thought many of them moved westward, including to south-east England. They were never dominant even before that, and Charlemagne annexed them in 785. For the next 300 years they were subjects of the various empires in north-western continental Europe. They funtioned as an intermittently autonomous state from 1100 by the grace of the HRE. They were finally annexed by the county of Holland and the city of Groningen in the 1300/1400s. Aside from a brief rebellion (1515-1523) they were subsequently subjects of the Saxons, the HRE, Burgundy and finally the Republic.

For the most part, they were simply too weak compared to their neighbours. They weren't wealthy nor numerous enough. Even nowadays Friesland and Groningen are kind of fringe provinces, too far north to really profit from trade and the accompanying presence of population centers.

It should also be noted that the Frisians aren't the Frisii, who were powerful enough to at least somewhat resist the Romans and also took part in the great northern rebellion.

There might be several explanations possible. And the answer iam going to give you is mainly from a work that is a little dated; "De lage landen bij de zee: Een geschiedenis van het Nederlandse volk" by Jan and Annie Romein.

In the early middle ages, the Merovingian and Karlongian times, we no longer hear about the big Roman trade ports in the low countries, for example Fectia (Vechten), Forum Hadriani and Trajectum ad Mosam (Maastricht). Those places who had a international trade, mainly as last Roman trade centres before northern Europe and as ports to trade with Britain. In those early middle ages we see a new important centre of trade appear in the low countries, Dorestad (Wijk bij duurstede). This centre of trade had the same function as the old Fectia, it was a throughput port for trade with Britain and over the Rhine river. This port was controlled by a people called the Frisians. These Frisians lived in an area what is now roughly: Zeeland, Holland, Utrecht, Friesland and Groningen. After the Karolingians reduced the Rex/Dux Frisia to counts; we enter the period of the County of Frisia. In the second half of the 9th century we have a small natural disaster, the Rhine becomes no longer easily navigable and the domination of Dorestad starts to wane. This is moment the Flemish cities, mainly Brugge starts to grow and take over part of the role of Dorestad. (Although we also see around this time athat some of the cities in the Northern-Netherlands become more trade oriented suchs as Utrecht, Tiel and Deventer.) Only much later during the 80 years war we see that the most important centres of trade shift north again. Then we see the fall of Brugge and Antwerpen, and the rise of cities like Vlissingen, Rotterdam, Delft, but mainly Amsterdam.

But to come back to the County of Frisia, this county that rose from the ashes of the Karolingian expansion, claimed to be the successor of the old Frisian Kingdom/Dukedom, but only ever really controlled Holland, Zeeland and Utrecht. Only around the year 1100 we see for the first time that this area is referred to as the county of Holland. And with good reason, the people living there might have spoken a different dialect, but they would have called themselves Frisians.

So to conclude:

• The people we in retrospect call Dutch, will in historical documents be called Frisians. And the idea of being Dutch was not present at the time, even after the formation of the county of Holland in the High Middle ages. That idea of a "Dutch" ethnic group is a more of a fabrication of the Dutch Republic.
• The decline of Dorestad as a major centre of trade will have facilitated a shift of trade to the Flemish cities like Brugge.

You are questioning the shift from Frisian dominance in the 8th century, to Dutch dominance in the 15th-17th centuries. You assume that Frisians became overshadowed in this period.

Frisian trade relied on the Frisian ship type, a rather innovative small fairly economical proto-cog. This ship was particularly well suited for the coastal North Sea. Frisian dominance then also relied on trade in this region: basically the stretch from Jutland (Ribe) down to the major river estuaries (Dorestad). Before these trading centers were founded, trade primarily took place between individuals or on temporary beach landings. In Frisia itself, by the 8th century a parallel specialist site-type was formed in the 'langwurten'. Instead of these central nodes (like Ribe and Dorestad), these seem to be more local, and more distributed rather than focused. With the foundation of the emporia, trade equally shifted from the road-in-between (the Frisian coastland, and the Frisian ships) to the endpoints. With the incorporation of Frisia into the Frankish sphere (and later the Scandinavian, under Viking conquest), it were these representatives of the endpoints who by the 9th century determined trade. Equally, Frisians had to choose in the Medieval period between a Baltic Hansa and a Flemish/Dutch Atlantic. The Frisian shiptype was equally eclipsed by the more efficient 'proper' cog, carrying more cargo, but you should keep in mind that this medieval shift was still founded not only on Frisian prototypes in trade models, but also ships and people.

So, in short, their dominance was based on their unique ability to sail the Wadden sea stretch, but as more universal shiptypes as well as political circumstances developed, the basis of this advantage fell away. Frisians had to adapt in consequence.

I am in a bit of a hurry the moment (moving houses), so it might take me a while to reply to followup questions. Be patient, though.