My guess is Charlemagne's conquest in the late 8th century: they'd earlier been venturesome enough to be considered one of the Germanic peoples to settle in England. But the Frisians of the 4th century onward seem not to be the Frisii of Roman times, instead merely inheriting the name of the area as later Germanic arrivals from the east.