The simple answer is that the bullets were traveling pretty slow, compared to modern ones, so they needed to be heavy to be effective. Until the invention of the Minié conical bullet, those muskets were also firing round balls. A .69 caliber round ball would be 480 grains, a .58 Minié 500 grains. So, a bigger caliber was needed to throw about the same weight ( or, more precisely, a conical bullet could be a smaller caliber). Like muskets, the military rifles were also large caliber, but unlike the muskets required a cloth patch around the ball, to act as a seal and help it grip the grooves of the rifling.

Conical bullets were also found to be more effective: they penetrated better, and had less air resistance and so had flatter trajectories. Not just in muskets. Before circa 1870, Rifles for hunting very large game, like elephants, would shoot very heavy "belted" balls , weighing perhaps 3 oz., or 1,312 grains. Hunters soon switched over to cartridges using conical bullets around 600 grains when they became available. Not only were they still effective, they didn't wallop their shoulders with as immense a recoil...shooting a 3 oz ball would create recoil about as heavy - or heavier- as both shooting barrels of a 12 ga. double-barrel shotgun, which is something that people avoid doing for very good reason.