George Washington and Benjamin Franklin were freemasons, which makes sense because they were an elite group.

This could not be more wrong! There's a (probably untrue) story of a reporter who was interviewing, then President, Washington being shocked to hear him call his gardener, "Worshipful Master," but whether it happened or not, it illustrates a truth that's quite powerful: Freemasonry is the union of the laborers and the ruling class in a common pursuit, with every effort made to discard those distinctions inside the Lodge.

In fact a concrete example of that persists to this day: white gloves are worn by officers, and centuries ago, by all members. This was used to disguise the rough hands of a laborer and the soft hands of a member of the aristocracy.

The freemasons were made up of mostly elite members of society, such as politicians, judges, businessmen, etc.

... and soldiers, preachers, authors, musicians, etc. There are many examples of these. " Free Masonry in North America from the Colonial Period to the Beginning of the Present Century," published in 1889, documents the charitable giving to poor and distressed Masons, as do many other historical works of the 18th and 19th centuries.

You have it backwards. Many of society's elite were Freemasons, but they were vastly outnumbered.

How come the Federalist Party's successor, the Whig Party, adopted anti-masonic sentiments?

Freemasonry had it's unpopular scandals, one of which (the Morgan affair) was the nominal reason for the anti-Masonic party's founding, but the real reason, IMHO, was the same as any political opposition: people who disagreed with the opposed group felt that it was in possession of too strong a position in public life. Many politicians were Freemasons, and that led to distrust of the Fraternity.

Wikipedia on the topic:

In New York at this time the faction supporting President John Quincy Adams, called "Adams men," or the "Anti-Jackson" faction, were a very feeble organization, and shrewd political leaders at once determined to utilize the strong anti-Masonic feeling in creating a new and vigorous party to oppose the rising Jacksonian Democracy. In this effort they were aided by the fact that Andrew Jackson was a high-ranking Mason and frequently spoke in praise of the Order.

You can read Adams' anti-Masonic screeds for detail on what the claimed failings of Freemasonry were.