What a great question! I specialize in Medieval History of England and France from roughly 1066-1215, and Brittany is a little bit of a special, side-interest for me.

The exact origin of the Ermine is not very clear. While Duke Jean III of Brittany made the ermine the official arms of Brittany in the early 14th C, it had been part of Breton heraldry before this time period. John V later went on to create a chivalric order called “The Order of the Ermine” in the 1380s. There are also stories that it originated with Anne of Brittany in the late 15th Century. Supposedly Anne saw an ermine surrounded by muddy terrain, and rather than dirty it’s white coat, the ermine chose death. Anne admired the creature and made it her emblem. However, this story is probably spurious, as there were already well-known connections between ermine and Brittany from centuries prior.

The origins of most heraldic devices are very murky. Because of the scarcity of visual sources and material culture remaining from the High Middle Ages, it often can’t be said with certainty that the first person known to use a certain image or color scheme was the originator.

I was not able to find anything that specifically attributes where the ermine came from, and we’ll probably never really know, but I did some looking at visual sources and found something interesting:

The Bayeux Tapestry depicts the Norman Conquest in 1066. Modern consensus is that it was commissioned by William the Conquerer’s brother, Bishop Odo of Bayeux, who is depicted several times in the tapestry itself, though popular myth says it was made by Queen Matilda and her ladies. However, it is known that it was constructed within living memory of the events. While it is a famous case of history being written by the victors, the fact that it was made shortly after the events lends some authority to certain elements of the depiction.

Now what does the Norman Conquest of England have to do with Brittany? Quite a lot, actually.

The first half of the Bayeux Tapestry does not actually depict William’s invasion of England, but the Breton-Norman War of 1064-5. This conflict brought Brittany into a sort of personal union or tributary status with Normandy. The importance of this event to the Tapestry is that supposedly Harold Godwinson was one of Duke William’s allies, and swore allegiance to him as the future King during this event. But we’re concerned with the Bretons.

Norman Knights laying siege to Dinan

Here we see a conflict between Norman Knights and Bretons. Looking inside the castle, we see two shields that are white, with an irregular pattern of spots, and a pattern of red among the defenders. From this conflict, William took many Breton nobles into his entourage, and many of them accompanied him on the Norman Conquest, as we see in this panel showing the ships crossing the channel. On these ships, we see the same white shield with spots and the red cross pattern. But we also see a plain white shield with spots.

We see this plain white shield a few more times throughout the tapestry, though the bearers are not specifically identified. Considering the detail to which some of the shields are rendered on the tapestry, it is reasonable to think that these were meant to identify certain people or groups. High profile figures would have expected to have their deeds attributed to them in such a public demonstration of the “official story”.

One candidate for a person depicted in the tapestry who has not been identified is Alan Rufus, a younger son of the Count of Penthièvre. One peculiarity of Hastings is that we actually know remarkably few of the people who were on the battlefield. Alan Rufus is among those who were almost certainly present. Alan was granted massive amounts of land by William in Cambridgeshire and Yorkshire, and was eventually given the Honour of Richmond. He died childless, and his titles passed through two younger brothers, a second Alan, and Stephen. Stephen of Treguier’s son, Alan “the Black” would eventually inherit the title himself. This Alan was married to the heiress of the Duke of Brittany, Bertha, and their son would eventually be Duke Conan IV of Brittany, from whom all the subsequent Dukes of Brittany descend.

This is largely circumstantial, but I find the correlation between the white shield with a speckled pattern associated with Bretons and the adoption of a w white flag with a pattern of stylized spots quite telling. Unfortunately, as the tapestry is not large enough for details, we cannot be certain if this is an early depiction of the ermine, if this is a pattern that eventually evolved into the ermine, or if we're seeing what we want to see. The purpose of heraldry at this time was to distinguish forces on the battlefield, and early heraldry often consisted of simple patterns, or large symbols in contrasting colors. It is not unreasonable that we are seeing a step in the evolution of heraldry here, but unfortunately we do not have the full story, only glimpses.

I’m sorry that there isn’t a straight answer to your question, but I hope you at least find this interesting and thought-provoking.