To build on /u/mosin91 's excellent answer, one has to realise that before the invention of guncotton and smokeless powder, black powder caused several problems for gun design.

Black powder burns pretty inefficiently, and you need a relatively large amount to send a bullet at decent speed. It also leaves a lot of residue - this is why black powder cannons where whisked with water - to remove any gunpowder residue as well as cool the barrel somewhat.

Thus, firing several shots in short order with a black powder weapon could clog the barrel and make the next shot explode in the barrel, killing or wounding the wielder of the weapon rather than his enemies.

There was generally resistance against equipping infantry with repeating arms, as the large amount of smoke after a salvo from a tightly formed infantry unit made it impossible to see the enemy - what good is it to be able to shoot again, if you cannot see the enemy? Cavalry was supposed to move a lot more and faster than infantry, and would thus be able to move out of the smoke. Thus the resistance against repeating weapons was far less in the cavalry.

That aside, firearms did improve over time.

Take for example a standard musket.

In the late 1400s, this was a large and expensive arqebuis. It was smoothbore, so heavy it required a fork to be held level when fired, was unrealiable due to bad metallurgy and dependent on good weather, as the matchlock could stop working in bad weather.

The musket became lighter, faster to load and soldiers started to carry pre-loaded amounts of gunpowder in bandoliers. Toeards the mid-1600s, the weapon was light enough to drop the fork. About the same time the bayonet was introduced, allowing musketeers to fight cavalry. First a plug bayonet inserted into the barrel, then a ring bayonet placed on the barrel, then a socket bayonet, often fixed to the wood stock to not bend or othersiw distrupt the barrel when fighting with bayonets.

Then the flintlock was introduced - this did not require a burning fuse to be attached to the musket and allowed muskets to be pre-loaded and the pan to be covered, making the weapon much more reliable in bad weather (although still not very reliable).

In the late 1600s, the paper cartridge was introduced - gunpowder, bullet and fatted paper in one, making the reload process easy. Just bite open the cartridge paper, pour the gunpowder down the barrel, ram the paper and bullet down, full your pan with gunpowder and you have a loaded musket.

In the mid-1700s, the Prussians introduced the iron ramrod. Before ramrods had been mostly made out of wood. Iron ramrods were not as prone to breaking and could be used with much more force.

Around the same time, rifled barrels were introduced in several European militaries. The Swedish light infantry in the Seven Years War used rifled carbines called studsare. Rifliing makes the musket far more accurate as it makes the bullet spin, but it also requires more force to push the bullet down the barrel and makes reloading slower.

By now, the rate of fire for a musket has improved from about 1 shot per minute to about 5 shots per minute (which both Austrian and Prussian training centres required of a recruit before introducing him to platoon training during the Seven Years' War). Increasing the volume of fire 400% is quite an improvement. Muskets had also become much lighter and wieldy, much more reliable, been supplied with bayonets and could be pre-loaded in a way that earlier muskets could not.

After the Napoleonic war, rifled muskets started to become standard and ball and cap started to replace the flintlock as the main firing device. The French also invented the minie ball, a more aerodynamic cone-shaped ball rather than the earlier round muskedball. This increased accuracy further.

While Gustavus Adolphus' infantry preferred to fire at ranges below 70 meters and the army of Charles XII of Sweden noted that not even sharpshooters could hit anything beyong 100 meters (and firing at 60 and 15 meters before charging before 1708 and 20 and 10 meters after 1708). The rifled muskets with minie balls could hit targets at about 600 meters.

Around this time muskets started to get more than just rudimentary sights since they could also shoot much further.

During the 1840s, the closed metal cartridge started to appear, making ammunition that could be loaded quickly and tightly (if gas escapes, the shot will not be nearly as fast when leaving the barrel). At the same time, machining steel started to become so prevalent that interchangable parts and gas-tight breechloaders became possible at a large scale.

In regards to your first question.

Designers were limited by the technology, metallurgy and ignition systems of the day. It's pretty hard to make a flint lock revolver or repeater that anybody would care about, but that didn't stop people from trying. Heck, the Puckle Gun was an early 18th Century gatling-esque firearm, that was ahead of it's time. And guns like the Ferguson Rifle tried to speed up the reloading time of the single shot rifle. In the end, it didn't become practical to start looking at breechloaders, or various repeating rifles until the advent of the percussion cap, and then the self contained metalic cartridge.

Now, as for the Spencer, it saw wide use in the Civil War, and was very popular, especially with cavalry. However, it's adoption was delayed by stubbornly conservative procurement officers, as well as cost. Of about 144,500 Spencers made during the Civil War, the US Government acquired 107,372, clearly nearly the entire output of the company.