Here in Canada, we hear a lot about the United Empire Loyalists, who left the American colonies during the war because of their loyalty to the British Crown. But we never hear about people (if any) who went the other way. Was there a movement of people we could call the anti-Loyalists?
I can't give you exact numbers at this moment, but in central Ohio there is a tract of land that was originally set aside for Canadian sympathizer refugees from the British Empire. This was an issue that the Congress took seriously, and the remnants of it are still seen in the names of roads and settlements in the area today.