how did European military powers coordinate different uniform colors during the time period when military uniforms were designed to be conspicuous on the battlefield?

Put simply, they didn’t. Uniform colours were at the whim of whatever ruler was in charge, technically, but matters of practicality and tradition also affected uniforms.

Surely there were some instances of armies meeting each other only to find an enemy wearing the same color.

Indeed, and as different branches of service within country's militaries often wore differing colours, confusion could result.

Uniform colours that began in practicality often became traditional: the red coats of the New Model Army, chosen because red was the only dye available in sufficient quantity, became established as being definitively British by the mid-18th Century. Others had their roots in politics: the tricolor uniforms - marrying the 'revolutionary' red and blue of Paris with Bourbon white - designed by Lafayette for the Garde Nationale in 1789 was extended haphazardly to the rest of the French Army during the revolutionary wars (hampered by the extremely shoddy state of French logistics in that period), and by Napoleon's rule had become simply what French soldiers wore, despite France being no longer really revolutionary or ruled by a Bourbon! Napoleon did experiment with turning some regiments back to practical, easy-to-wash white in 1809, but this was hampered by the outbreak of war with Austria, so the scheme did not come to anything much.

Often, earlier in the century, individual colonels had a lot of leeway in how they distinguished their particular regiments; before Choiseul's reforms of the French Army in 1762, the choice of uniform colour was wholly up to a regiment's colonel. Which side a given unit was on was sometimes only distinguishable by their 'colour' (the flags they rallied around, not the colour of their uniform); typically, each country would have its own kind, sometimes also carrying a national flag (as in British service) or having a distinctive ornamentation (like French Imperial Eagles).

There was nothing stopping two armies outfitting their troops with very similar uniforms. At the Combat of the Coa in 1810, Portuguese gunners in the fortress of Almeida mistook British riflemen, dressed in dark green, for French troops (French Dragoons and Chasseurs a Cheval, who in the Iberian Peninsula sometimes went unmounted, wore green) and fired on them. Similarly, in 1807 one French brigadier (the cavalry equivalent of a corporal, not the general of a brigade) on an intelligence mission utilised the similarity of his blue uniform to Prussian ones to bluff his way past a party of Cossacks to retrieve a parish priest who might give the French valuable intelligence. At one point, a red-coated Swiss unit was fired upon by its British enemies, but was then also fired upon by its French allies as it retreated.

Many different arms of individual armies dressed differently: French infantry of the line wore blue and white with red ornaments; French light infantry wore mostly plain blue, whereas large parts of the French cavalry wore green. British infantry wore the famous red coat, but British cavalry (and, later, artillery) commonly wore dark blue, and of course the Rifles were famously greenjackets. The Generally, it was the job of officers to discern, in the chaos of battle, who was on which side, using uniform ornamentations and a units flags and cadences to work out whether or not they should be shooting at a given unit, although usually this was clear enough given that your enemies would likely be attacking you.

This is, of course, all complicated by the fact that the nominal colour of a unit's uniforms might not be the actual colour of a given soldier's coat on campaigns. This is an era before modern synthetic colourfast dyes, so redcoats in the Peninsula usually ended up a pinkish colour with liberal patches of brown Portuguese cloth. The survivors of Napoleon Bonaparte's Egyptian expedition returned to France in 1801 in a rainbow of colours, from green to violet. Further, officers would usually order uniforms for their men dyed much darker than the 'regulation' colour, so they would hold their colour better and fade to the regulation colour on campaign - thus 'the dark green of the chasseurs [a cheval] of the Guard is almost black, the sky blue of the 5th Hussars is almost royal blue" [from Elting, p441].

In short, then, there was no real coordination, and which armies used which colours was always a combination of politics, practicality, and tradition, which could at times cause real confusion in the field.

Sources:

Rifles: six years with Wellington's legendary sharpshooters, by Mark Urban

Redcoat: The British Soldier in the Age of Horse and Musket, by Richard Holmes

Swords Around a Throne: Napoleon's Grande Armée, by John R. Elting