Although officers, particularly in the cavalry, did still receive swords during the American Civil War (ACW), they were largely ceremonial. When they were put to use, it was usually in conjunction with a pistol as a means to parry a bayonet thrust (usually during a charge). This was the case for both the infantry and the cavalry, and although the officer corps of both branches did use their swords to rally troops or indicate the direction of an attack, they were rarely used for direct, confrontational engagements with an opponent.

As OP mentioned, a gaggle of pistols tucked away on a cavalryman's person and saddle rigging was the preferred method of going into battle for those on horseback, especially for Confederates, who rarely stayed put long enough to need a rifle. The Union was a bit more flexible in this regard, and did put dismounted cavalry into action at times (Gen. Buford's 1st division at Gettysburg probably the most famous example of this). Yet the practical realities of fighting on horseback rendered rifles mostly irrelevant for these troops, who were designed to scout, screen, scatter disorganized infantry, and/or harass an enemy's flanks. This made swords even less valuable to the cavalry, since the pistol provided all the killing potential as a sword, yet with vastly increased range.

Now, don't get me wrong: swords were used on horseback, and men were killed with them. It's just that these casualties paled in comparison to those shot by cavalrymen, who would have only used the sword if out of ammo, or unable to quickly get to their revolvers.

[Sources: Bruce Catton, 'Grant Moves South'; Shelby Foote, 'The Civil War: A Narrative, v. 2'; Douglas Southall Freeman, 'Lee's Lieutenants, v. 2']