I'm not a Medievalist, nor do I study heraldry, so I can't speak to the extent that this was an issue, or how it was dealt with in all cases, but I can provide some brief, illustrative examples to say that it was likely an issue, and it was one which would be litigated.

One alleged case arose during the First Crusade with two knights, one from Holland and the other from Tyrol, encountering each other and bearing the same coat of arms. A dispute quickly arose and it was submitted for adjudication by the commanders. As it turned out though, it seems that both had sufficient proof to argue their case, which meant that the assembled court couldn't pass judgement. As such, they were directed to settle their dispute by wager of battle. John van Arckel, the Hollander, triumphed, slaying his opponent, and thus retaining the rights to bear the heraldry in the Holy Land.

Now, while this is a singular case, it would suggest a pretty standard way in which these disputes were settled. Had one knight been able to prove his ancestral right to the crest with considerably more proof, the matter would have been settled by the assembled court, and he the winner of the case. The case instead resulted in wager of battle as there was no clear winner, which is the circumstance in which you would expect to see the use of the ordeals - to settle matters in which an allegation cannot be proved by the available evidence.

That being said though, even though is possibly fits the mold in which Wager of Battle would be utilized, there is some questioning about the extent of its use at the very least. A similar case, arising in Scotland in 1312 where an Englishman, Hugh Harding, successfully bested the Scotsman Walter de Seintlowe, is called greatly into question, being the lone case of a dispute regarding Heraldry in the Kingdom of Scotland being decided by force of arms. George Nielson, in his classic "Wager of Battle", finds the resulting Writ from King Robert to be a forgery, quite possibly created for Harding who was attempting to create a few tales of daring-do to his name. It is attested that disputes would be settled by a knightly duel in the Kingdom of England, but evidence for this only can be found from the reign of Edward III onwards, who took the throne in 1327, and as part of the 'Duel of Chivalry', which while similar in relation, not quite the same as Wager of Battle.

To be sure, I don't point the above out to necessarily call into question the first account provided. The use of the wager of battle differed through Europe, after all, but I am saying that the answer is probably somewhat unsatisfying for you. A lot of "maybe" and "perhaps". We can confidently say that disputes of the crest of arms happened, and that they had to be settled. We can also say with confidence that some of those disputes likely ended with an appeal to arms, but nothing I have would give me confidence in saying how often it happened, as the sources we have are not always the most trustworthy!

The van Arckel episode is recounted in "Superstition and Force" by Henry Charles Lea. Harding and Seintlowe in George Nielson's "Trial by Combat". Both are quite out of date, but peppered with great anecdotes such as this. For a more recent information on the Wager of Battle, check out "Trial by Fire and Water: The Medieval Judicial Ordeal" by Robert Bartlett.

There were a number of ways.

One was simple heredity: you didn't choose a conflicting coat of arms because you bore your father's coat of arms, or perhaps your arms impaling your wife's arms, or quartering the coat of arms of your parents, depending.

There were heralds, originally an offshoot of troubadors. Among their duties were to know coats of arms, especially as they were helping to run tournaments.

When meeting in a tourney or joining an army, heralds would sometimes compile "occasional rolls [of arms]", which would give a further chance to conflict-check. A king of arms of a tourney could require a participant to add a mark of difference or otherwise deface their arms during the tourney.

In England, there was the Court of Chivalry to judge cases pertaining to arms and originally to military discipline in general. (Well, technically, there still is, but the judgment in the Manchester Palace of Varieties Ltd case in 1954 said that, if there were to be more cases in the future, they'd be a lot more comfortable if this old prerogative court were placed on a statutory basis. An interesting brief overview of English regulation.) There were other courts, like of constables or marshals on campaign.

From the 1530s in England, there were visitations of arms: kings of Arms and heralds travelling about the country, though they were mostly looking for assumptions of any arms without licence, much less conflict. (Actually, they seemed to be mostly looking for fines.)

Another way to handle it was evade the issue entirely by asserting that arms needed to be unique only in a jurisdiction. Scrope v Grosvenor was a famous early case. They met on campaign. Scrope won the case because he proved that his family had borne the arms since the Norman Conquest. (Which was impossible, as that was roughly a century before the invention of heraldry. The suggestion I've seen is that Scrope was particularly politically important.) Carminow came along and proved that his family had borne the same arms since King Arthur, but his claim of conflict was dismissed for political reasons because Cornwall had originally been its own kingdom.

I forgot to note: I have the impression that crests (the things on top of helmets), as opposed to coats of arms (the things on the shield), were generally informal and pretty unregulated. But I'll gladly defer to anyone who has more information.