The innovation happened in steps.

First, breech-loading guns replace muzzle-loading guns. More than anything else, this happened from military necessity: observers noticed that when breech-loading Dreyse rifles went up against muzzle-loading ones, in Bismarck's European wars, breech-loaders won. That kicked off a rush in the 1850's. Some of those breech-loaders used paper cartridges, however- and so they couldn't have magazines ( well, some ingenious people tried that and they didn't work very well, or were very expensive)

Second, metallic cartridges are developed. Now, it's possible to have bullets carried in the gun. The most prominent example of this would be the Henry rifle, which Confederates complained could be "loaded all Sunday and fired all week". Like the Winchesters that followed, it had a tubular magazine. The Winchester Model 66 did as well: the Swiss adopted that tubular magazine into their Vetterli, in 1869, the Germans refitted their 1871 Mauser with it in 1874. Other tubular magazine guns, like the Kropatschek and the Lebel would follow.

Tubular magazines were slow-loading, however. And, when smokeless powder ammo came in, the higher-velocity smaller caliber bullets were more pointed...and in a tubular magazine, that bullet point-against primer configuration was bad. So, spring loaded box magazines were introduced ( the 1886 Mannlicher) . Then a very ingenious man, James Paris Lee, came up with a box magazine that was removable, had a double-column of cartridges, and ( very importantly) could be quickly loaded with charger or stripper clips. Lee's magazine was a part of the new 1889 British rifle, which was named the Lee Metford. That rifle was the one that replaced the Martini-Henry that had been used at Rourke's Drift.

Why everyone didn't just jump ahead to box magazines in 1855 is a good question. We could also ask why there weren't armed flying drones in 1980. When the British adopted the Lee Metford, the US was making plans to finally abandon the single-shot black powder Trapdoor Springfield. Sometimes technology doesn't leap.

Assuming you're referring to the period where metallic cartridges were widely available and rifle magazines had been proven, the problem was largely doctrinal. There was a huge amount on resistance by army leadership among the Great Powers towards the adoption of a magazine rifle. The main concern was logistics combined with poor training - there was fear that soldiers would fire off all their ammunition in a panic if they had a magazine rifle, and excessive expenditure of ammunition was a major concern in an era where armies remained heavily reliant on the same pack animals that had been the mainstays of logistics for centuries once they got away from their railheads. The concern was particularly important for colonial- or frontier-oriented armies like those of the US, Britain, or France.

As armies started to wise up to the utility of a magazine in a rifle, we start to see limited experiments and adoption of magazine rifles starting with areas where the logistical concerns were less of an issue. The Swiss, with their smaller army and defensive focus, adopted a magazine rifle in the form of the Vetterli in 1868, and the Kropatschek rifle would see adoption by Austria and France in more limited marine roles (as the marines were expected to be fighting either from ships or very close to them, so ammunition would be less of a concern) and as a general-issue rifle by Portugal. By the 1880s, armies were starting to wise up to the utility of a magazine rifle, so we start to see conversions like the German Gewehr 71/84 or Dutch Beaumont-Vitali.

One thing to note with all of these early magazine systems was that they all lacked a speed loading system. Up until the invention of en-bloc and stripper clip systems, magazine rifles in theory had no faster fire rate than a single-shot rifle - after all, a magazine rifle that's expended all of its ammunition still has to be single-loaded to refill the magazine. Speed loading systems were rapidly adopted and became the standard once they were invented, but their absence definitely played into the thinking about the limited utility of magazines.

Once smokeless powder came around, the logistical concerns of ammunition supply were mitigated further, as the small-bore cartridges were significantly smaller and lighter and thus easier to transport. At the same time, a lot of the old thinking about wastage of ammunition was still prevalent among leadership - although leaders accepted the utility of magazine rifles, a lot of armies insisted on the inclusion of a magazine cutoff. Magazine cutoffs were a system installed in the rifle to allow the soldier to keep the magazine disengaged and the gun functioning as a single-shot until directed to engage the magazine by an officer. These systems were very prevalent, appearing in the Lebel and Enfield among many others. The Americans included magazine cutoffs in both the Krag Jorgensen and the considerably more refined Springfield 1903.

Really it wasn't until the realities of WW1 set in and wartime demand for rifles encouraged streamlining production of rifles as much as possible that the magazine cutoff concept really died. Combat experience was showing that logistics were more than capable of keeping up with the fire rates afforded by magazine rifles, and meanwhile magazine cutoffs were extra parts, machining, and materials that could be easily foregone to speed up rifle production.

Clarification question for OP, if that's okay: Are you referring specifically to detachable magazines? Magazine is a general term for anything that stores ammunition, including ship's compartments or even buildings. The kind of magazine in older rifles you described I think would be an internal magazine.

I know firearms terminology can get pretty pedantic, sometimes needless so (see silencer/suppressor) but felt a distinction should be made here. I also just call them magazines most of the time.