I think it's a stretch, without a firm source, to say the revolutionaries would have definitely been hanged for treason if the war went poorly. If the land-owning, formerly-respected members of the revolution sued for peace, there could have been a possibility of some manner of pardon, depending on each side's respective position in the war and their desire to conclude the war on favorable terms.

For example, formal discussions between revolutionary leaders and the British took place on the subject of ending hostilities -- such as the Staten Island Peace Conference, where John Adams and Ben Franklin met with Lord Howe after the rebel loss in the Battle of Brooklyn. So I don't think a negotiated peace is out of the question, even though it proved untenable at that point.

But we could say that Franklin and Adams were able to meet with British authorities without being arrested on the spot. They were treated as legitimate negotiators and recognized as such, at least during the period when they had troops in the field. If those troops were decimated, British reception would naturally be less hospitable, but the question is whether the revolutionary leaders would have sued for peace after defeat became apparent, but before the British destroyed their ability to wage war. That question seems impossible to answer. We know the rebel leaders had a "do or die" attitude at the start of hostilities (paraphrasing Patrick Henry), but that was before they had established themselves as enough of a political and military annoyance to merit negotiation.

Another fact of note is that in August 1775, King George proclaimed that any colonist who took up arms was engaged in treason, yet this was not enforced in the actual conflict in the following years. Alan Valentine's biography of Lord Germain notes that despite his command, British generals did not hold treason trials and instead held captured rebels as POWs, contrary to official royal decree. The capture of numerous British troops at Saratoga provided additional incentive not to hang captured rebels.

This is not to say that rebel prisoners were treated with hospitality. Far from it. See, e.g., Lang, Patrick J., "The horrors of the English prison ships, 1776 to 1783, and the barbarous treatment of the American patriots imprisoned on them." But what it does show is that the British were not fully committed to treating the uprising as a treason issue, at least so long as military and political considerations were at play.

Interesting footnote - In 1990, lawyers from the US and the UK held a mock trial in a British court where they tried George Washington for treason. The court found Washington not guilty of all charges. Somehow I doubt that an 18th century UK court would be as forgiving, but again, it's hard to say exactly what terms a Colonial surrender would entail, depending on the strategic situation.

EDIT: A bit of checking shows that John Adams learned many years later that his name was on a list of people specifically excluded from any offers of pardon Lord Howe might make to the rebel leaders. See Edgar, Gregory T., Campaign of 1776: the Road to Trenton (1995). So that might be the best answer. Pardons were contemplated, but not for all.

In Washington: A Life (Pulitzer Prize for Biography 2011), Ron Chernow says that George Washington planned to flee to his lands in the Ohio Valley that he acquired between the French and Indian War and the Revolutionary War. Chernow didn't go into much more detail than that--such as how he expected to stay safe once there, which Washington probably hadn't considered either--but that was his plan.

It is highly likely that the British would have tried them with treason, particularly those which signed the Declaration of Independence, and if that happened, they would most likely be put to death. I forgot which of them, but I know a few of the original signers made a point to sign in large print, as an affront to the King, as they figured if they lost they would die no matter what. The colonists probably would have faced more taxation and further military occupation.