Partially the idea stems from the Romans themselves. For example, the Romano-British historian Gildas, writing in the sixth century, states that the Anglo-Saxons who raid and conquer England following the Imperial withdrawl, are sent as a punishment from God for the sins of the Britons, chief amongst which was homosexuality.
Invasion-as-divine-punishment is a frequently used instrument in the toolbox of early medieval authors writing as much to instill Christian morality in their audience as they are to actually chronicle or educate. Just as Gildas says the Anglo-Saxons are divine punishment, so Bede says about the Vikings. To Gerald of Wales, the Norman Conquest of 1066 was necessary to punish the sins of the Welsh and the laxity of the English Church, and the Anglo-Norman invasion of Ireland in 1172 was determined to end the shameful sins of the Irish and bring them into line with the English church.