The best smidge of evidence I'm familiar with that says he could comes from 1751^1.
3 Dec. 1751.
Ann Carrol and Mary McDaniel Servts [servants] of Fredericksburgh, being Committed to the Goal [jail] of this County by William Hunter Gent., on Suspicion of Felony, & Charged with robing the Cloaths of Mr George Washington when he was washing in the River some time last Summer, the Court having heard Severall Evidences Are of Oppinion that the said Ann Carroll be discharged, & Admitted an Evidence for our Lord the King Against the said Mary McDaniel.
And Upon Considering the whole in Evidence, & the prisoners defense, the Court are of Oppinion that the said Mary McDaniel is Guilty of petty Larceny, whereupon the said Mary desired Immediate punishment for the sd [said] Crime & relied on the Mercy of the Court, therefore it is ordered that the Sheriff carry her to the Whipping post & Inflict fifteen lashes on her bare back, And then she be discharged & [etc].
Sometime in the summer of that year he took off his clothes and got into the Rappahanock River for what the court says was "washing" but in all reality was probably an effort to cool off and/or take a quick swim. While he was in the river two women supposedly robbed his possessions from his clothes, one being found guilty and given 15 lashings for it. Washing at that time was more like a sponge bath or freshening up - hands and face were washed in a basin in your chamber or at the shoreline of a stream. Total submersion for cleansing was a rare thing and most colonists were fearful that too much of it could easily disrupt your balances (hot, cold, wet, and dry had to be held equally else you'd get sick, though Dr B Rush did later recommend to bathe in cold water to cleanse the skin and promote even sweating in the warmer months, but that was a while later), which is a big part of why baths weren't a common thing. Cooling in a river, though, was much more common. That taken with the fact that he was so far removed from his clothes that he was unable to either a) know about and/or b) stop his stuff being taken shows he was likely going for a swim to cool off on a hot July day. He didn't testify directly in the trial, unfortunately, because at that time he was already on a ship headed to the Caribbean with his brother (where he would catch smallpox and nearly die). All this happened while he was a late teenager.
The average colonists would not have been able to swim. One man who wasn't by any stretch of the word average or ordinary could, and quite well. In fact few common people today would be capable of performing the aquabatics and feats of swimming accomplishment that this man did, and you already know his name: Benjamin Franklin. "Living near the water, I was much in and about it, learned early to swim well, and to manage boats..." He could see the masts of the ships docked at Boston, smell the salty air, and hear the noises of the docks and shipyards all from his house. Mill Pond was also near his home and the waterways surrounding the town were his playground. A prepubescent Franklin engaged in an early civic project of building a levy with stolen stones from a nearby house being built to enable his mates and himself a better fishing spot. He also devised a "sail-kite" that would pull him across the pond with the breeze. He was flying his kite, then staked it to take a quick swim. Finding the swim enjoyable he was resistant to get out, but wished to continue flying his kite. Deciding it wasn't an either/or, he got out, unanchored the kite, and returned to the water while holding it. Soon his feet were lifted and he laid on his back, then found himself gliding across the water "without the least fatigue and with the greatest pleasure imaginable." His first legitimate inventions in a long list of contributions to mankind were specifically for swimming and came not long after the sail-kite, and one is still very widely in use still today: flippers/swim fins. His weren't as effective as the sail-kite namely because they were made of wood - the rigid foot flippers didn't replicate a fish tail and the more effective hand paddles were hard on the wrists.
This wasn't the end of his swimming. Even in his late teens and early 20s he continued to swim regularly, particularly in the Thames while in England and once swam from "near Chelsea to Blackfriars, a distance of at least two miles"^2 in which he claims to have been "performing on the way many feats of activity, both upon and under water." On the return voyage to America he would dive from the ship, swim around it once or twice, then climb back on for a quick thrill, something that would likely terrify any of us to even consider doing in the same transatlantic sailing situation.
He didn't stop until physically unable; in his 70s he was still teaching others the art of swimming. A large part of his comfort was his understanding of water and sickness - he felt fresh air and activity were key to health. He didn't buy into the whole "wet hair makes you catch a cold" notion which allowed him to spend several hours in the water at a time, something he was right about yet still a very common thought in America until not long ago.
Spotsylvania County Order Book, 1749–55, p. 141
Benjamin Franklin, Edmund S Morgan, Yale Univ Press (2002)
Also used for sourcing was The First American: The life and times of Benjamin Franklin, H.W. Brands, Anchor Books (2002), the best biography I've read of the Good Doctor Franklin.
Most Franklin quotes here are from his autobiography.
E for typos