I have often heard that most sailors during the Napoleonic Wars could in fact not swim. However, i have read from various sources that French (in particular) Light Infantry, such as the Voltigeurs on several occasions swam across rivers under fire from the enemy. If Infantrymen could swim then surely a large portion of sailors should've been able to swim as well?
Edit: Example from a source originally published in 1828:
“the French arrived [at Tordesillas], 60 ... headed by Cpt Guingret, a daring man, formed a small raft to hold their arms and clothes, and plunged into the water, holding their swords with their teeth, swimming and pushing their raft before them. Under protection of a cannonande they crossed this great river, though it was in full and strong water, and the weather very cold, and having reached the other side, naked as they were, stormed the tower: the Brunswick regiment then abandoned the wood, and the gallant Frenchmen remained masters of the bridge." (Napier - "History of the War ..." Vol IV, p 138)
A combination of practical and cultural factors were at work here, which have been addressed in some earlier responses:
Is it true that a lot of old-timey sailors couldn't swim?
Sailors ability to swim in the 15th century
You are correct that there is no obvious reason why soldiers should be able to swim better than sailors, however, and I would suggest that the river fording activities that you mention very likely involved a combination of wading and swimming behind makeshift rafts or other floating objects. This is surely heavily implied also because troops would need a way of carrying their arms with them in order to engage the enemy once the river had been crossed.
I asked a very similar question a couple of weeks ago and didn't get any response. From some recent reading I've done, it seems that civilian sailors in the years during and directly following the Napoleonic wars had rather high levels of swimming ability. There are two accounts I've come across of small boats being wrecked that I'll repeat.
In 1811 the discoverer of Campbell Island, Captain Hasselborough, left the ship in a small boat with five other people to inspect oil casks the sealing party had left. On the way back to the ship, 'upwards of a mile' from the shore in subantarctic waters the boat was capsized. The captain was wearing boots and a heavy boat cloak, and sank almost immediately, though, as this fact is made much of, we should presume that the average sailor could swim ok in better conditions. George Allwright, a 12 year old boy from Sydney, also sank fairly quickly, though his age may have been part of this. A sailor, Richard Jackson, and 'a New Zealand [i.e. Māori] boy' whose name and age aren't recorded, made it to land ok (though clearly the Māori boy is not relevant to the question). The ship's carpenter, James Bloodworth, who was clearly quite a strong swimmer, attempted to help the captain and the boy, but, after failing, swam for shore supporting the last occupant of the boat: a young woman named Elizabeth Farr from the convict settlement of Norfolk Island. She was a good enough swimmer to keep herself afloat while he tried to rescue the other two: but tired quickly, and, by the time he got them to shore, had drowned. She was also likely quite young: the couple of people I've seen mention her follow the original report and the Norfolk Island censuses in referring to her simply as a 'young woman', but this author has found a birthdate for her that would make her 14 (and makes the question of why she was even on the ship a bit troubling). Of the three that drowned then, two were probably children, while the third was uncharacteristically heavily dressed. Both the adult male sailors could swim over a mile in bad conditions.
In 1820 the Missionary Register repeated a story told to them by Tooi (Tui?) a Māori guy who was in England, about his time working on a whaler. According to him, a whale smashed their boat, and broke the captain's legs. The other four sailors swam to a whale they had killed previously: 'about two and a half miles off' and then trod water there until 'Tooi', who had been helping the captain, arrived with a knife for them to cut steps on the whale to get on top of it.
Again, these are only two stories, but in both of them the sailors seem to be fairly good swimmers.