My understanding is, by the time of the American Revolution, rifles existed but they were so slow to load that they weren't efficient in battle. However, by the time of the American Civil War, those same soldiers would have consistently used rifles. What were the reasons for the change?
My frame of reference here is the United States, but the question is obviously the broader European world.
The Minnie ball was the change. In order for a rifle to work, the round must be large enough that the rifling in the barrel can engage it and make it spin
In smooth bore muskets, the round does not fit that tightly into the barrel. That is why muskets used a patch around the ball. The patch held the ball in the musket, but didn’t prevent it from leaving the musket upon firing.
In older rifles, the bullet had to be almost exactly the diameter of the barrel. This made them very hard to load. It was so tight the the bullet either had to be pounded in with mallet or screwed into the barrel.
The Minnie ball changed this. It was conical bullet that was smaller than the diameter of the barrel, making it easy to load from the muzzle. The back of the round had an opening that extended about 2/3 the length of the round. When the rifle fired the opening in the back of the round allowed it to expand until it engaged with the rifling of the barrel, causing it to spin.
As a result, the Minnie ball was as easy to load as a musket ball. However it would be able to expand into the rifling and fly further, with greater accuracy, and greater force than a smoothbore musket ball.
.
Two important developments.
First, a flintlock was loaded in a few steps. For the military musket, a soldier carried cartridges in a cartridge box. There were a charge of powder and a lead ball wrapped in paper. The soldier would typically bite onto the ball through the paper and tear open the cartridge. Some of the powder would go into the pan of the lock, for priming, the rest would go down the barrel, then the ball would be rammed down the barrel with the paper on top, to keep it from rolling out ( though, after several volleys, the bore would foul enough to make it unlikely the ball would roll out, anyway.
This meant there was both a somewhat loose fit of the ball in the bore, and that the powder charge varied. Accuracy is the same as consistency: the gun does the same thing over and over again, so that it's possible to predict, when aiming, where the ball will hit. If the ball can come out of the bore in various ways, and the powder charge varies, that's impossible. Rifles, that would also spin the ball and could have more accuracy, would first require a measured powder charge and a patch-wrapped ball. So, loading one took longer, and faster rate of fire then ( as well as now) was considered more important.. There is a common misunderstanding of this point: if a gun could not be first loaded consistently, rifling itself would not help accuracy.
The first improvement was the caplock, circa 1830. Instead of flint striking steel and setting off a priming charge in the pan, a percussion cap would detonate the powder charge. This meant that the powder charge in the cartridge was no longer divided: it all went down the barrel, and so pre-measured charges could be consistent.
The second improvement was the Minie ball, or bullet, circa 1850. It was a loose fit, could be rammed home down the barrel, but would expand on firing, making a tight fit. Minie bullets therefore didn't need to be patched, still left the barrel more consistently than loose balls. The Minie bullet would also grip rifling grooves, and because of that and with the better consistency in ammunition, rifling could then add to the accuracy. Rifles then began to replace muskets in the 1850's. With increasing numbers of effective mass-produced breechloading guns around the same time, though, the Minie bullet itself soon became obsolete