A major reason for people fighting in dense formations was, essentially, cavalry. I would argue that it took until the Crimean War before infantry in line formation were (using minnie ball rifles) able to put out sufficient weight of fire to stop a cavalry charge at range. Even then this was considered dangerous and remarked upon with the 'Thin Red Line' at the Battle of Balaclava being a prime example.

That incident involved two hundred British infantry and about 350 Turkish troops being charged by roughly four hundred Russian cavalry. They had time to fire three volleys before the Russian troops broke off the charge and would perhaps have had the opportunity for a final, fourth shot at point blank range if the charge had been pressed home. The first volley was at a range of 600 yards.

Note that this still involved a two man deep line standing shoulder to shoulder rather than a more dispersed formation as you might see with modern infantry.

If you look back even a few decades further to the Napoleonic wars you see that armies were almost uniformly using unrifled muskets with a similar rate of fire but much, much shorter range, perhaps one hundred yards effective reach. This meant that there was normally only time for one or at best two volleys against charging cavalry. Skirmisher troops using rifles fired even more slowly due to the laborious reloading process prior to the introduction of the minnie ball.

If you lack the density of fire to stop cavalry from riding into your troops with lances or swords then you need tight formations to prevent them from, after charging, engaging in one sided slaughter. Something similar applies to other infantry, until firearms technology advanced enough a more dispersed formation was always potentially vulnerable to being attacked in close quarters by a dense, disciplined block of other infantry then cut down in hand to hand combat.