So we're all on the same page, here's the part of the Constitution in question: Article II, Section I, Clause 5

No Person except a natural born Citizen, or a Citizen of the United States, at the time of the Adoption of this Constitution, shall be eligible to the Office of President; neither shall any Person be eligible to that Office who shall not have attained to the Age of thirty five Years, and been fourteen Years a Resident within the United States.

The Heritage Foundation has a very interesting webpage about various details of the Constitution. Being a Conservative think-tank, their Constitutional guide has a somewhat originalist/strict constructionist bent. In practice, this means they often include details of how the framers thought of various articles. Here's what they have to say about the eligibility requirements in Article II:

Second, the President must have been a "Resident" of the United States for fourteen years. By contrast, to be a Member of Congress, one must be an "Inhabitant" of the State one is representing. During the Constitutional Convention, James Madison contended that "both [terms] were vague, but the latter [‘Inhabitant'] least so in common acceptation, and would not exclude persons absent occasionally for a considerable time on public or private business." Then as now, inhabitant meant being a legal domiciliary, but resident could mean either a domiciliary or a physical presence. Perhaps the Framers desired a person as President who had actually been present in the United States for the required period and had developed an attachment to and understanding of the country, rather than one who was legally an inhabitant, but who may have lived abroad for most of his life. On the other hand, the distinction may have been one of style rather than substance. As Justice Story later noted, "by ‘residence,' in the constitution, is to be understood, not an absolute inhabitancy within the United States during the whole period; but such an inhabitancy, as includes a permanent domicil in the United States."

There is some evidence that the Framers believed the fourteen-year residency requirement could be satisfied cumulatively, rather than consecutively. An earlier version of the clause excluded individuals who have "not been in the whole, at least fourteen years a resident within the U.S." (emphasis added), and historical evidence suggests that deletion of the phrase "in the whole" was not intended to alter the provision's meaning. This might explain the election of Herbert Hoover, whose successful 1928 campaign for President came less than fourteen years after his return to the United States in 1917. Others may argue that Hoover had simply maintained a United States domicile throughout his tenure abroad.

They mention no clear reason why it's 14 years rather than, say, 10 or 15. It seems, like the "natural born"ness of the president's citizenship, like it was meant to ensure that the presidential candidate have a physical tie to the U.S. and didn't have divided loyalty.

As a follow up question, Did John Adams have any political enemies try to use this against him? He spent nearly 8 years from 1778 to 1788 in various ambassadorial roles to France, Netherlands, and England. Arguably he hadn't been a permanent resident of the US during that time. So did anyone try and argue that?

As described in James Madison's notes on the Federal Convention of 1787, what would become Article II, Section I, Clause 5 was presented on September 4th as part of a report by the third Committee of Eleven, set up on August 31st to discuss various issues that had previously been postponed. The clause was agreed to on September 7th without any dissent, but unfortunately also without any other comment from Madison.

But a month earlier, on August 9th Gouverneur Morris (representative for Pennsylvania and a frequent and active speaker in the Convention, who was subsequently on said Committee) had moved that 14 years of citizenship should be the qualification for Senators. Rufus King (also on the Committee) in his notes described Morris making the following argument:

Foreigners cannot learn our Laws or understand our constitution under fourteen years. Seven years are requisite to learn to be a shoemaker, & double this Term will be necessary to learn to be an American Legislator; and it will require at least fourteen years to eradicate native attachment & the affections of Education.

So Morris was rhetorically framing a fourteen year period as a sort of necessary citizen apprenticeship when he first proposed it (though in the case of Senators this was ultimately not approved by the convention).

As /u/yodatsracist pointed out, the language that the Committee originally proposed had the phrase "in the whole, at least fourteen years a resident", and the debates on qualifications for Representatives on August 8th featured concern from some over whether the wording might

"exclude persons absent occasionally for a considerable time on public or private business"

or

"exclude men, who had once been inhabitants, and returning from residence elsewhere to resettle in their original State...a want of the necessary [local] knowledge could not in such case be presumed" (i.e. someone could have lived elsewhere for a while but still be suitably familiar with the region's issues).

Admittedly, in this case they were discussing Representatives rather than the President, but their expressed concerns about residence were more focused on someone spending enough time to be familiar with a place, rather than it being strictly consecutive. In the end they didn't choose to make a point of clarifying it in the clause.