In movies like Brave heart and such, all the fighters look the same and yet you see them almost instinctively knowing who is an enemy and who isnt. Was this accurate? or was there a lot of friendly fire?
Don't rely on Hollywood to give you anything approaching a reliable recreation of pre-gunpowder mass combat. What keeps you from killing your buddies in battle is what also keeps you from getting killed by the enemy: Keeping in formation.
Swirling melees are terrible, and rare, occurrences because you can't always tell who's who, unless everyone is uniformly equipped. Instead, two opposing formations (shield-walls, boar's heads, etc.) would meet and, generally speaking, the warriors involved would keep on trying to murder the warrior who was facing them. Or maybe their buddy to the side or behind them. Either way, they were facing their foes.
Once formations broke, the victors would be able to quickly and easily kill the people fleeing the fight, so it boiled down to kill the people who are in front of you and running away.