Because they were shooting round ball, propelled by slow-burning black powder. Unlike modern bullets, which are almost universally conical - that is to say, they are much longer than they are wide and taper to a point - round balls have certain weaknesses inherent to them. A round ball in any given caliber is going to be much lighter than a conical bullet in the same caliber. In order for the balls to be heavy enough to effectively penetrate, it's necessary to increase the diameter; this also helps increase range, as a heavier projectile is more resistant to wind than a very light one. It's not that they were using too much gun - it's that, given the limitations of the technology (single shot, slow reload, low muzzle velocity, inefficient shape) small calibers weren't viable for military use.

Part of it is that to the mind of early gun users big was better, the slow-burning black powder limited velocity so going bigger was the way to get more energy. A Larger bores is easier to load and you can use undersized shot or buck and ball.