Can't speak to artillery, but for small arms there has always been a balancing of a few factors: range, lethality, and weight. In the 19th c., after the introduction of metallic cartridges, velocities were low and to compensate and still get adequate muzzle energy, bullet weights were rather high- lots of them in the 11mm to .50 caliber range ( and the previous muzzle-loaders had had even larger calibers, with even lower velocities) . But at 200 meters something like 45-70 , with a big 400 grain bullet, would drop around half a meter- which was an arc steep enough to make it hard to accurately fire at long ranges, even with elevated sights. When higher-velocities came in with smokeless powder circa 1885, it was possible to get better range and similar muzzle-energy from lighter bullets, and something around 7 mm/30 caliber became standard- longer range, lighter ammunition, equal lethality. Just before WWi it was assumed that longer range was a real advantage, but after WWII, it was realized that most small arms were used at under 100 yards and rate of fire was more important- which meant that ammo weight was more of a consideration ( soldiers had to carry more rounds to feed their rifles), thus the .223/ 5.56mm became more standard; a small bullet, but going very fast.

This was not an orderly, neat process of discovery, innovation. The Swedes, Italians and Japanese decided that smaller bullets ( 6.5mm) at smokeless-powder velocities were just as effective , despite their shorter range; then the Japanese changed their minds and went to a larger cartridge that was better in their machine guns, the 7.7MM The Germans used 8mm , something slightly bigger.

For handguns, great range was never much of a consideration. Somebody like Elmer Keith could write about hunting with a pistol at 200 yards, but officers shooting their pistols at that range would not have been effective or useful. There was therefore perhaps more variation in caliber- at the small end, 7.62X25 was a lighter bullet going fast, at the upper end , 45 ACP was a heavy bullet going slow, 9mm Parabellum was in between. But the muzzle-energy for all three was around the same- 390, 365 and 351 ft/lbs.

If you pick up a copy of Barne's Cartridges of the World, you will notice there are groupings of calibers around 7.62/.30 caliber for big game and military, .23/5.56mm calibers for high-velocity shorter range hunting and military use, 9mm/45 for pistols. You will notice that there are also lots of cartridge designs for these calibers....pages and pages, many of them ( Barnes often noted) quite redundant. Like wood screws and lawn mowers, there are a bunch of different designs that all work OK, but because none work perfectly someone will have always been introducing a new one.