There's no real evidence that people in Brittany saw the conquest as payback for the Anglo-Saxon invasions. (There's also much reason to doubt whether the invasions actually happened, though many believed they had by the 11th century.)

A few years after the conquest, however, this started to change. Geoffrey of Monmouth wrote a history of England (c. 1136) in which he claimed to have discovered a bunch of old Welsh stories about King Arthur's wars against the Saxons. Whether or not these Welsh stories were real or something Geoffrey made up to improve his credibility, Geoffrey made them - and their hero, Arthur - an important part of the story of England. Modern historians analyzing Geoffrey's work agree that he did this in order to ingratiate himself with the Norman aristocracy (his principle audience), because making Arthur the greatest English king, and the Saxons his enemies, was great politics. By taking out the Saxons, Arthur's enemies, the new Norman elites could be argued to be restoring, not upsetting, the proper order of things in Britain.

Before Geoffrey, we only have scattered references to Arthur and his battles in a few early medieal texts. But after Geoffrey's history, Arthur exploded in popularity to become one of the most important pseudo-historical characters in high medieval fiction.

It's safe to say that, by the end of the 12th century, many would have recognized a connection between Brittany, Arthur, and ancient grudges between the Saxons and British that needed to be avenged. But whether this memory was alive before - wheter those songs and folk stories Geoffrey claimed to be basing his stories on were real, and whether such stories were known in Brittany as well as Wales - is something about which we can only guess. There's no evidence I know of beyond speculation to suggest a popular imagination of exile among the Bretons before Arthur stories became popular in 12th century Norman France.