The simple answer is gas pressure/large potential for backfire and producing a gun cheap enough and designed conveniently enough to do so.

The longer answer is: From a naval artillery perspective, an advantage so far overlooked in this discussion of breech loading artillery is being able to load a gun on a ship without either going outside the ship where the barrel end is or pulling the gun back inside the ship. A number of early guns on ships were not set on carriages with wheels, they were made stationary in their assigned position. There are some examples of these on the Mary Rose. Breechloading made it a lot easier to load, not having to expose oneself while reloading it. The 16th century wasn't the same as the rest of the Age of Fighting Sail. They were trying out a variety of guns, ammunition, positions to mount guns, and so on since the role of warfare at sea was quickly escalating. The Mary Rose has such a variety of different guns that demonstrates this - it wasn't like the Ships of the Line that later had set specifications of X guns on X deck in X caliber.

But by far the reason breech loading guns (the large ones in particular) didn't catch on for a long time was the gas pressure and backfire issues. When you used one of these breechloading guns, what they do is they have a breech piece that is shaped a lot like a modern mug. The charge for the shot goes inside this mug-like breech chamber, it's set back into the breech, and then...here is the dangerous part...the breech chamber has to be wedged up against the rear of the barrel, it's not a perfect fit and lock kind of situation. The chamber has to be wedged enough to take the recoil from the gunpowder ignition and not allow the chamber to come back enough and allow gas to escape or create an outright backfire. In the heat of battle, it is possible to forget to wedge the chamber properly, ignite the charge, and then get a huge explosion in your face and even injure/kill the gun operators. On bigger guns, this is approaching an insane risk at times considering the size of the charge. The reason this type of gun continued to be used (besides them being passed about and various places short of weapons using them since there was no other choice) is that the smaller ones (the swivel gun versions particularly) were probably a bit more manageable (easier to make sure it's wedged, the backfires impacting the one operator and not a crew of several, and its likely happening on the upper deck and not below decks, which complicates things even more). It wasn't until the 19th century that a better design for breech loading came about, at which point the industrial revolution allowed the safer design to be mass produced at a cost effective price.

Infantry weapons didn't use a similar mechanism because, again, the gas seal issue was really hard to resolve until the 19th century (and imagine trying to implement the breech chamber concept on that small a firearm, at that point it would be easier to just muzzle-load it). The few breechloaders that did exist before the mid nineteenth century were quite expensive, were prone to issues, and didn't take off in use during the Revolutionary or Napoleonic Wars. The clincher to Infantry use of breechloading weapons on a large scalle was the ability to mass produce a breech-loading weapon design that was safe, easy enough to operate, and could be massed produced at an affordable rate (though a select few generals in the American Civil War thought that giving infantry multiple shots like that would just waste ammunition, so there was the tactical issues on the battlefield to contend with as well, having officers trained in a method that emphasized gunfire by individuals and not massed volleys).

That's what I got. It's interesting how my two specialties, Age of Sail and American Civil War, really worked together well to inform on both ends of this issue.