What did they use before heraldry was widespread?
More can always be said on the matter, of course, but as this exact question of combat identification is one of the more common warfare questions on the sub, here's the same question from last week, answered by u/J-Force and also with my usual post compilation about Medieval combat identification.
I'll try and offer a perspective from South Asia, here heraldry didn't play much of a role in battles, and yet the opposing armies managed to make heads and tails of who was friend and who was foe. But before we dwell on that let us take a look at how battles generally progressed in the time period of our concern.
Based on what we know of how medieval combat went down, you most probably wouldn't be in a scenario where combatants are locked in individual melee, where lines are mixed or when the ranks of each side meld into one another, losing any cohesion.
What happened, most likely, and based on the sources we have, is that two lines advanced but when they got to a distance where they would soon meet, the front and second and perhaps even the third ranks facing the enemy would meet, only the first ranks engaging in actual fighting, the ones in the immediate rear giving them space to maneuver and watching their flanks and the ones in their rear still, holding position and waiting with anticipation.
The understanding of warfare in the period as it exists today, tells us that in battle, no more than the first and second ranks engaged. When two opposing armies, or two opposing infantry formations approached one another the advance would be in accordance with a pace at which the soldiers could maintain cohesion and support of their neighbouring comrades, collective action and support in depth from the men standing behind the front rank soldiers in deep files, gave the men in the front ranks confidence. Once the enemy came closer the front ranks slowed their advance and the men standing behind the front rank in deep files would follow suit, eventually the first ranks came into contact, which the second rank stopped enough paces behind the first so as to allow for the first to be able to fence and fight. The job of the second rank was to support the first rank, to check his flanks, to avoid enemy break through or sudden pushes and to replace the soldiers standing before him should the man in the first rank be slain or injured by taking his place and allowing him to be dragged to move to the rear.
Keep in mind, not to ignore the human nature of war. Of morale, of fear, of organisation and the necessity of discipline. Warfare was never an all out push or melee with combatants mixed up in enemy ranks lacking any sense of direction or understanding of whether they were fighting the enemy or their own comrades, but rather a test of nerves, skill and a period of ebbs and flows of combat. The lines of infantry were never straight, but rather curved or had bends depending on where the enemy had gained or lost ground in account of prowess in arms or exhaustion of units that gave ground. Front ranks on each side would engage and at times disengage in period of lulls in violence. The front most rank would be at the centre of the battle, the recipient of all its tensions, vigours, fears and desperation, while the ranks in their rear would await in anxiety, nervousness and restlessnes. By acting as units, by standing in deep files and columns, men were able to, by virtue of organisation and discipline, shake off to a certain extent, that fear and that concern for self preservation. This is why such formations existed. To tame men's hearts and to give them a decided advantage over unorganised forces, where men fought as individuals.
Now, there are also instances when lines weren't as disciplined, or when the combatants were in solid formations, but rather more spaced out on one side or both to facilitate a shock tactic. Of charging at the enemy with ferocity, to break their morale and then slay them with superior feats of arms.
There's an intance which comes to mind, of an encounter between an army led by Trilochanapala, Brahmin King of the Kabul Shahi against the army of Mahmud, son of Sabuktegin at The Battle of Chachh, 1009 CE
Firishta remark at how a corps of "6,000 archers was sent to provoke the enemy by Mahmud, in response to which roughly 30,000 Ghakkars, charged the Muslim lines, with axe, spear, halberd, bow and arrow and sword. Within an hour of fighting roughly 5000 Muslims lay dead"
Even here, the Hindu Ghakkars, charging the Muslim line, didn't lose sight of friend and foe. The reason being that such charges had a practical thinking behind them. To charge a line only to have a few soldiers in the front ranks break through, surrounded and slaughtered by enemies from all sides was suicidal. Hence, such charges involved causing impact and chaos at the enemy front and break the enemy from the front, cause a loss of morale and a break in the enemy's spirit and cohesion, to force him to give ground and eventually rout, which would allow for the slaughter to begin.
Sources :
"Battle Studies" by Col. Ardant Du Picq
"The Hindu Shahis of Afghanistan and Panjab" by Yogendra Mishra, pg 146-150.