I have not checked how much Benjamin Franklin's opinions changed in later years, but at least in 1751, Benjamin Franklin spoke even of Russians, Swedes and most Germans as being swarthy and therefore not white.
This is especially baffling when he happily considered the "English" (unsure how narrowly he is using this term) and Saxon Germans to be white.
What factors would have led him to regard most Europeans as non-white in 1751 solely on the basis of their skin complexion?
For easy reference, see paragraphs 23 and especially 24 (the last paragraph) of his essay "Observations Concerning the Increase of Mankind, Peopling of Countries, etc.":
http://www.columbia.edu/~lmg21/ash3002y/earlyac99/documents/observations.html
Also pasting the relevant paragraphs in question as a comment:
"And since Detachments of English from Britain sent to America, will have their Places at Home so soon supply'd and increase so largely here; why should the Palatine Boors be suffered to swarm into our Settlements, and by herding together establish their Language and Manners to the Exclusion of ours?
Why should Pennsylvania, founded by the English, become a Colony of Aliens, who will shortly be so numerous as to Germanize us instead of our Anglifying them, and will never adopt our Language or Customs, any more than they can acquire our Complexion.
Which leads me to add one Remark: That the Number of purely white People in the World is proportionably very small. All Africa is black or tawny. Asia chiefly tawny. America (exclusive of the new Comers) wholly so. And in Europe, the Spaniards, Italians, French, Russians and Swedes, are generally of what we call a swarthy Complexion; as are the Germans also, the Saxons only excepted, who with the English, make the principal Body of White People on the Face of the Earth.
I could wish their Numbers were increased. And while we are, as I may call it, Scouring our Planet, by clearing America of Woods, and so making this Side of our Globe reflect a brighter Light to the Eyes of Inhabitants in Mars or Venus, why should we in the Sight of Superior Beings, darken its People?
Why increase the Sons of Africa, by Planting them in America, where we have so fair an Opportunity, by excluding all Blacks and Tawneys, of increasing the lovely White and Red? But perhaps I am partial to the Compexion of my Country, for such Kind of Partiality is natural to Mankind."