TL;DR: Improvements in metallurgy, allowing for smaller grain sizes and higher chamber pressures, along with various other improvements that increased the ballistic efficiency of guns and allowed less propellant mass to achieve equal muzzle velocities.

#<Warning: probably a bit physics-heavy >

I can provide some insight in the physics behind cannon barrels and ballistics, to provide some general explanations. I don't know the developments of the period by heart however, so I will add the historical context and overview of relevant developments in the 18th century a bit later when I have more time.

Ideally, the muzzle velocity of a projectile is determined entirely by only 3 variables:

1. Force on projectile (along a section of a barrel)

2. Mass of projectile

3. Barrel length

This is still quite simple, and quite useless. In practice, the force of the expanding gases on the projectile is not constant along the barrel, and it's much easier to work with the pressure times the cross-sectional area (often expressed as the bore of the gun) of the projectile instead. Of course gun barrels and projectiles are not infinitely strong, thus mechanical properties such as yield and ultimate strength of the projectile and barrel must also be taken into account. The pressure in the barrel is created by a chemical reaction, the speed of which is mostly dependent on the type of compound used, the mass of compound used, and the grain size.

This already leaves us with a whole bunch of key variables that can be changed to get different shapes and types of chemical guns, ranging from the tiniest pistol to mortars with ridiculously short and wide barrels. Summed up the key variables to muzzle velocity are now:

1. Projectile mass

2. Projectile ultimate strength^^1

3. Barrel length

4. Barrel thickness^^2

5. Barrel material yield strength

6. Propellant compound

7. Propellant mass

8. Propellant grain size^^3

9. Bore radius / projectile cross-sectional area

This is a bit more complex as you can see, but with these variables we can accurately model the characteristics of any conventional gun. Now onto applying it to actual early artillery!

Early cannons were very limited in range^^4, minimum mass, minimum bore size and propellant, due to the low-quality cannon material. Cannons came in many, many, many shapes and sizes, but commonly were made out of cast iron or cast bronze. Being in its infancy, the quality of the cast guns was relatively terrible and they contained many faults that compromised integrity. This meant pressures inside the cannon had to be kept low to prevent the cannon from turning into a giant bomb. Keeping the pressures low means less force on a projectile and correspondingly less velocity, which can only be compensated for by increasing barrel length, or increasing bore size. And that's exactly what was commonly done. The bombard for example had proportions that look utterly ridiculous to us with a very wide bore and very short barrels. The wide bore ensures however, that a projectile can be accelerated fast even with a relatively low pressure^^5. Not unimportant either, is that a short barrel reduces weight considerably, which allows for a thicker barrel, which allows for higher pressures. You'll see where this is going: There are a ton of trade-offs to make, and everything is tied together. You change X, then Y and Z change as well.

So what changed around the 17th century that allowed for the use of less propellant mass?

The 'ballistic efficiency' increased, likely due to better metallurgy and other continuous developments. Ballistic (BE) efficiency is defined as:

Ballistic efficiency (BE) = Kinetic Energy of projectile / Energy of propellant

This is a percentage, often between 1-50%.

A lot of gunpowder is being burned in a cannon, but not all of the energy that is released is being used to accelerate the projectile. Early cannons had terrible BE, which meant a lot of gunpowder had to be used to achieve the desired results. The most important reasons for this were the aforementioned low chamber pressure, the short barrels, and something known as 'windage' - the gap between the projectile and the barrel of the gun. Lacking precise manufacturing tools, the windage of old cannons was very large and reduced the already low BE even more. Cannons were continuously being improved over this period however. Metal casting techniques, as well as other aspects of metallurgy were all being developed and resulted in cannons that could support higher pressures at the same mass. Stronger materials also allowed for the use of finer grained propellant. Gunpowder with a small grain size burns much faster, which increase acceleration of the projectile for the same amount of propellant mass, and thus increases BE (only up to a certain point though). Improvements in the precision of barrel and projectile manufacturing could reduce windage, which also increases BE. Lastly, a longer barrel will always improve BE, and thus we can predict that this may have been the case.

I'm out of time right now, and I think this will be enough of a general physics answer, but I'll see to comparing some specific examples of the period later on and checking which variables changed the most. It's unlikely to be just one though; more likely the general design and manufacturing of artillery improved, involving several variables.

Addendum:

1. The reason this needs to be taken in account is that the projectile can and will be destroyed if the pressure exceeds its ultimate strength. Barring a few exceptions, having a projectile break up is bad. Stone projectiles could be used in early cannons for similar reasons. Chamber pressures and projectile acceleration were low, which meant the stone projectiles stayed intact despite being exceedingly brittle compared to even low quality iron. As chamber pressures increased, stone projectiles could no longer sustain the stresses and disappeared, as cast iron projectiles eventually did as well.

2. The exact pressure a barrel can handle is of course a more complicated result of its geometry and composition. For understanding this, it is however enough to see that increasing the chamber pressure of a gun requires either a thicker barrel or a stronger material. The former of which increase weight, which is a serious consideration, and the latter of which requires an improvement in metallurgy.

3. Grain size is the size of individual grains of gunpowder. The technique for creating these grains was in Europe discovered centuries before the 1600, and it was thus widely known that changing the grain size also changed the burn rate inside the cannon. Cannons typically used (and still use) very large grains compared to rifles, because a rapid-burning propellant would result in destruction of such a large gun. Smaller grain size equals higher pressure, even without increasing the propellant mass.

4. Range is a function of muzzle velocity and the ballistic coefficient of a projectile. In this period, the aerodynamics of cannon balls were typically 'horrible' and not subject to much change. In addition to muzzle velocities being low compared to contemporary artillery, I will use range as a synonym to muzzle velocity here, consciously neglecting drag effects and such.

5. Only up to a certain point. An oversized bore will require more propellant in turn, to produce enough gases to fill the larger volume, hence there is an optimum bore size. There also more general downsides to having a large bore, most notably the ballistic coefficient of your projectile goes down (e.g. it's less aerodynamic for the same mass), so range can be reduced ^(not too relevant to the guns of the era though). Different sizes of projectiles also have vastly different characteristics on hitting their target, and having a projectile that is too wide and light may result in a dramatic lack of penetrating power ^(also not very relevant to the era).

TL;DR: Improvements in metallurgy, allowing for smaller grain sizes and higher chamber pressures, along with various other improvements that increased the ballistic efficiency of guns and allowed less propellant mass to achieve equal muzzle velocities.

#<Warning: probably a bit physics-heavy >

I can provide some insight in the physics behind cannon barrels and ballistics, to provide some general explanations. I don't know the developments of the period by heart however, so I will add the historical context and overview of relevant developments in the 18th century a bit later when I have more time.

Ideally, the muzzle velocity of a projectile is determined entirely by only 3 variables:

1. Force on projectile (along a section of a barrel)

2. Mass of projectile

3. Barrel length

This is still quite simple, and quite useless. In practice, the force of the expanding gases on the projectile is not constant along the barrel, and it's much easier to work with the pressure times the cross-sectional area (often expressed as the bore of the gun) of the projectile instead. Of course gun barrels and projectiles are not infinitely strong, thus mechanical properties such as yield and ultimate strength of the projectile and barrel must also be taken into account. The pressure in the barrel is created by a chemical reaction, the speed of which is mostly dependent on the type of compound used, the mass of compound used, and the grain size.

This already leaves us with a whole bunch of key variables that can be changed to get different shapes and types of chemical guns, ranging from the tiniest pistol to mortars with ridiculously short and wide barrels. Summed up the key variables to muzzle velocity are now:

1. Projectile mass

2. Projectile ultimate strength^^1

3. Barrel length

4. Barrel thickness^^2

5. Barrel material yield strength

6. Propellant compound

7. Propellant mass

8. Propellant grain size^^3

9. Bore radius / projectile cross-sectional area

This is a bit more complex as you can see, but with these variables we can accurately model the characteristics of any conventional gun. Now onto applying it to actual early artillery!

Early cannons were very limited in range^^4, minimum mass, minimum bore size and propellant, due to the low-quality cannon material. Cannons came in many, many, many shapes and sizes, but commonly were made out of cast iron or cast bronze. Being in its infancy, the quality of the cast guns was relatively terrible and they contained many faults that compromised integrity. This meant pressures inside the cannon had to be kept low to prevent the cannon from turning into a giant bomb. Keeping the pressures low means less force on a projectile and correspondingly less velocity, which can only be compensated for by increasing barrel length, or increasing bore size. And that's exactly what was commonly done. The bombard for example had proportions that look utterly ridiculous to us with a very wide bore and very short barrels. The wide bore ensures however, that a projectile can be accelerated fast even with a relatively low pressure^^5. Not unimportant either, is that a short barrel reduces weight considerably, which allows for a thicker barrel, which allows for higher pressures. You'll see where this is going: There are a ton of trade-offs to make, and everything is tied together. You change X, then Y and Z change as well.

So what changed around the 17th century that allowed for the use of less propellant mass?

The 'ballistic efficiency' increased, likely due to better metallurgy and other continuous developments. Ballistic (BE) efficiency is defined as:

Ballistic efficiency (BE) = Kinetic Energy of projectile / Energy of propellant

This is a percentage, often between 1-50%.

A lot of gunpowder is being burned in a cannon, but not all of the energy that is released is being used to accelerate the projectile. Early cannons had terrible BE, which meant a lot of gunpowder had to be used to achieve the desired results. The most important reasons for this were the aforementioned low chamber pressure, the short barrels, and something known as 'windage' - the gap between the projectile and the barrel of the gun. Lacking precise manufacturing tools, the windage of old cannons was very large and reduced the already low BE even more. Cannons were continuously being improved over this period however. Metal casting techniques, as well as other aspects of metallurgy were all being developed and resulted in cannons that could support higher pressures at the same mass. Stronger materials also allowed for the use of finer grained propellant. Gunpowder with a small grain size burns much faster, which increase acceleration of the projectile for the same amount of propellant mass, and thus increases BE (only up to a certain point though). Improvements in the precision of barrel and projectile manufacturing could reduce windage, which also increases BE. Lastly, a longer barrel will always improve BE, and thus we can predict that this may have been the case.

I'm out of time right now, and I think this will be enough of a general physics answer, but I'll see to comparing some specific examples of the period later on and checking which variables changed the most. It's unlikely to be just one though; more likely the general design and manufacturing of artillery improved, involving several variables.

Addendum:

1. The reason this needs to be taken in account is that the projectile can and will be destroyed if the pressure exceeds its ultimate strength. Barring a few exceptions, having a projectile break up is bad. Stone projectiles could be used in early cannons for similar reasons. Chamber pressures and projectile acceleration were low, which meant the stone projectiles stayed intact despite being exceedingly brittle compared to even low quality iron. As chamber pressures increased, stone projectiles could no longer sustain the stresses and disappeared, as cast iron projectiles eventually did as well.

2. The exact pressure a barrel can handle is of course a more complicated result of its geometry and composition. For understanding this, it is however enough to see that increasing the chamber pressure of a gun requires either a thicker barrel or a stronger material. The former of which increase weight, which is a serious consideration, and the latter of which requires an improvement in metallurgy.

3. Grain size is the size of individual grains of gunpowder. The technique for creating these grains was in Europe discovered centuries before the 1600, and it was thus widely known that changing the grain size also changed the burn rate inside the cannon. Cannons typically used (and still use) very large grains compared to rifles, because a rapid-burning propellant would result in destruction of such a large gun. Smaller grain size equals higher pressure, even without increasing the propellant mass.

4. Range is a function of muzzle velocity and the ballistic coefficient of a projectile. In this period, the aerodynamics of cannon balls were typically 'horrible' and not subject to much change. In addition to muzzle velocities being low compared to contemporary artillery, I will use range as a synonym to muzzle velocity here, consciously neglecting drag effects and such.

5. Only up to a certain point. An oversized bore will require more propellant in turn, to produce enough gases to fill the larger volume, hence there is an optimum bore size. There also more general downsides to having a large bore, most notably the ballistic coefficient of your projectile goes down (e.g. it's less aerodynamic for the same mass), so range can be reduced ^(not too relevant to the guns of the era though). Different sizes of projectiles also have vastly different characteristics on hitting their target, and having a projectile that is too wide and light may result in a dramatic lack of penetrating power ^(also not very relevant to the era).

In short yes, all three of those played a factor. Despite being an endless source of fascination for physicists like Galileo and Issac Newton, gunnery for a very long time remained more of an art than a science, relying on experience and reckoning to figure out what worked best.

Even putting aside the difference between corned powder and serpentine (which was still sometimes used even towards the end of the 16th century), there are a huge number of factors that affected its performance; the quality of the ingredients and manufacturing process, varying ratios of sulfur, charcoal, and saltpeter, as well as the size and shape of the grains could all dramatically affect the rate at which gunpowder burned. To add another complication, the "ideal" powder would vary depending on the specific dimensions of the weapon and weight of the projectile. If a specific powder burns too slowly, the ball will have left the barrel by the time it's finished, if a powder burns too quickly on the other hand, it will have an extremely high peak pressure when it is first ignited, but again less power overall. During the 16th and 17th centuries there were occasional mentions of improvements in gunpowder quality, and some manuals recommended using a different quantity of powder per shot depending on whether it was "good" or not.

One of the biggest changes occurred in the mid-18th century with Bejamin Robins and his invention of the ballistic pendulum. When struck by a cannon or musket ball, Robins could use the known mass of the pendulum and the maximum angle reached when struck to accurately calculate the velocity of a high-speed projectile in flight for the first time. This allowed him and other physicists to better study how different quantities of gunpowder affected muzzle velocity and develop equations to describe the internal pressures which occur when firing. One major development which came after a long period of extensive testing at England's Royal Powder Mills that began in 1783, was that charcoal produced in sealed iron chambers could produce gunpowder up to twice as powerful as it was traditionally.

The other major development which came about from Robins' 1742 New Principles of Gunnery was that by shooting the pendulum at different ranges he could study and calculate the effects of air resistance on both supersonic and subsonic projectiles, which had long eluded physicists despite having such a large effect on gunnery. His measurements of muzzle velocity showed for example that a 24-lb shot fired from a cannon should have been able to travel 16 miles assuming a parabolic trajectory, while in reality the maximum range was less than 3 miles due to air resistance. This lead to the creation of both far more accurate artillery tables for artillery officers (including the effects of both the gun's elevation and different charge sizes) and a better understanding of the effects of high velocities. In fact Robins went on to publish an article titled "A Proposal for Increasing the Strength of the British Navy" in which he argued for the adoption of lighter, larger caliber guns that used less powder and velocity, but could be made safer, easier to load, and better at smashing wooden ships with their large cannonballs without sacrificing too much range and power. The most notable example of this sort of thinking was the introduction of the carronade during the Napoleonic wars, but it would continue to influence cannon design and ship armaments for much of the 19th century.

Some Further Reading:

"Muskets and Pedulums: Bejamin Robins, Leonhard Euler, and the Ballistics Revolution" by Brett D Steele

The Gunpowder Age, China, Military Innovation, and the Rise of the West in World History by Tonio Andrade

Weapons and Warfare in Renaissance Europe by Bert Hall

A Proposal for Increasing the Strength of the British Navy