Ships in the later Middle Ages absolutely used flags, banners, pennants, streamers and painted sails to identify themselves, sometimes all in combination or sometimes individually. The Anglo-Flemish fleets worked in concert in the late 13th century, and agreements signed in 1286 and 1297 spelled out identification signals so that the fleets could recognize one another. (Frustratingly, the source that I'm looking at only mentions that, not what they were.) In the early 15th century, the English proposed that the Flemish ships would paint their home ports' coat of arms on their bows, though it's not clear this was carried out.

Admirals would of course have flown a flag starting at some point in the early Middle Ages (there's a reason we call them flag-officers), although keep in mind at this point "admiral" was a position rather than a permanent rank. If the admiral was sailing under royal orders, or if he was sailing a king's ship (rather than his own), they would often fly a royal flag or paint the royal coat-of-arms on their sails, which at the time were quite large -- the illustration exaggerates the scale of the men on board, but not by much.

Recognition at night was always more difficult, but admirals would carry masthead lights in particular configurations to distinguish their ships. We also know that there was a flag used to summon captains aboard the admiral's ship for a council of war, and that there seems to have been a flag signal to indicate an enemy in sight (the most ancient naval signal seems to have been a ship letting its sheets fly, to indicate an enemy nearby); but neither of those flag signals approached the complexity of those that started to be developed during the Anglo-Dutch wars in the 17th century.

Pennants, particularly red ones, seem to have been used to signal warlike intent. In May of 1293, a fleet from the Cinque Ports, including ships from Bayonne and Ireland, ambushed and attacked a Norman fleet near the Point de St. Mathieu. The Cinque Ports fleet seems to have been engaged in simple piracy, but an inquiry in 1299 found them defending their action by claiming the Norman ships were wearing bausans, a slender red pennant that they said traditionally meant that no quarter would be asked or given.

It's worth pointing out that while flag signals could be used to roughly coordinate movement at sea, or at least identify ships to one another, most of the naval combat in this period would need to be coordinated with armies on land. The English campaigns in Scotland and Wales were supplied by a steady stream of ships, and notably the French supplies during the siege of Bourg in 1407 were shipped up the Gironde. The The Bordelais captured a French convoy in the estuary and thus lifted the siege.

Sources:

• Jill Eddison, Medieval Pirates: Pirates, Raiders And Privateers 1204-1453

• Thomas Heebøll-Holm, Ports, Piracy and Maritime War: Piracy in the English Channel and the Atlantic, c. 1280 - c.1330

• N.A.M. Rodger, The Safeguard of the Sea: A Naval History of Britain, 660-1649