My wife and I have been watching alot of the history channel show "Vikings" and we love it. We often find ourselves asking, how the heck do these guys fighting on the front lines know who to hit next? Typically, in shows/movies like this you see two massive armies rushing towards each other. Initially when they make first contact, I can understand they know who they are fighting. But 5 minutes passes and how do you know who to hit at this point? Certainly not all armies had uniforms or some sort of visible markings on each soldier of their faction. Just something we found interesting..
I haven't seen the show Vikings, but I imagine from this question it follows the usual route of melee warfare by having it dissolve into a large melee with warriors from both sides having little personal duels all over the field. You're right that it would be extremely chaotic and friendly fire would be a massive issue, which is why Warfare typically wasn't conducted that way.
If you've ever seen a show focused on the history of the battle, there's probably a part where the camera zooms out to a topographical view and both armies are represented by colored blocks that move in unison. That was how it was done. Everyone joined together into a big colored block.
Jokes aside, a battle where everyone paired off and had one on one duels would be a horrendous slaughter, and nobody much wanted to die, so the best chances of survival would be fighting in formation, typically lines. That allows your archers and cavalry to function without fear of slaughtering your own men, and makes identifying enemies fairly straight forward in most situations. If they're facing you, they're the enemy, because all your mates are at your side or back.
For a better and more indepth discussion of identifying friendlies, you should consult this comment by u/WARitter, who does a great job of explaining the history of identification in the middle ages, as well as a notable time that friendly fire did happen.