I'm guessing it has to do with accuracy and/or availability, but I would like some professional input. I just watched the movie "Gettysburg," and observed how the infantry used rifles that took ~20 seconds to load, while the officers and cavalry used revolvers that could fire multiple shots rapidly.
Like most things in history, it's a combination of factors.
Cavalry were expected to be able to fire from horseback, which itself severely limited the accuracy of the gunfire. Maneuverability was key, and so shorter weapons were used like carbines and revolvers. These, especially the revolvers, had much shorter effective range. Revolvers could be expected to be accurate to around forty to fifty yards. Long guns like the Model 1861 Springfield, on the other hand, frequently doubled that figure.
The speed increase wasn't quite as beneficial, either. The revolvers used at the time didn't use shells like our modern-day revolvers. Instead, each chamber would be individually packed with powder and ball and loaded with a percussion cap. Grease would be smeared over the chambers to prevent the ignition of one from blowing up the entire pistol. The cavalryman or officer would thus be able to discharge four to six shots quickly, but then spend much more time reloading.
The revolver ended up belonging essentially only to the cavalryman. Almost all were manufactured by Colt. Even limited to cavalry, supply was an issue. The majority of pistols wielded by cavalrymen were older single-shot models, though naturally revolvers would have been preferred. Interestingly, officers received no formal training in the use of the revolver, and it was extremely rare that one should be used by an officer in battle.
For more information, I recommend Joseph G. Bilby's Civil War Firearms: Their Historical Background and Tactical Use.
A recent post regarding the use of muskets in the Civil War probably answers your question. There were several types of rifles that were superior to the musket but were not adopted. Difficulty of manufacture was one cause. Another reason was a belief that soldiers would waste ammunition if the rate of fire was increased. When breach loading and repeating rifles were used in the Civil war they were spectacularly deadly.
To add to the reasons already provided a revolver is an expensive and fragile weapon compared to a musket with more moving parts that must be exactly fitted by a skilled craftsman.
Supplying thousands of line troops with two would have been difficult at the time and maintaining them in working order likely impossible. A musket is a robust simple arm that is less likely to get out of sorts that can shoot a larger projectile a greater distance.
Also in the heat of battle all some troops can handle is powder, ball, primer, boom! powder, ball, primer, boom! powder, ball, primer, boom! Trying to load a revolver in the middle of a day long slug-fest fight with all the terror and confusion of the battlefield would be problematic.That said guerrilla groups like Quantrill's Raiders used cavalry armed with multiple revolvers to great effect.