I know some of the bloodiest combat happened after formations broke

The bloodiest combat would happen after the formation of one side broke, and almost all of the blood would come from them.

but were there rules to how you fought with the enemy, i.e. one-on-one.

Don't do it because it's stupid and suicidal?

The most basic, fundamental rule that governs battles before the gunpowder era (and it's mostly true of later eras too) is that if you have two friends around you and the guy you are facing doesn't, he is going to die and you are going to live. Because of this, almost everything about fighting is about how to ensure that you can maintain the safety of a formation.

Hollywood movies often show battles as first having organized lines which then break down into a disorganized general melee. This is completely wrong, and would never happen. The moment your line fails, you don't continue fighting the enemy, you either do everything in your power to rally our friends and form a new line, or you flee the field. Because your enemy is certainly going to try to rebuild his line, and the moment they succeed, they will completely roll over you.