I recently finished reading a book entitled The Rise and Fall of the American Whig Party, by Michael Holt, which goes into the breakup of the Whigs, although not the Federalists. There were two factors here - internal factionalism, and the rise of issues which they failed to properly integrate into their platform.

The obvious big issue was slavery. Whiggery had revolved around an interventionist state to promote economic improvement and commerce, a certain moral attitude of middle class Protestant piety, and a view that the legislature rather than the executive should lead public affairs. Northern bankers who wanted to lend money to creditworthy merchants and southern plantation owners who wanted a good market for their cotton and loans to tide themselves over in bad times could agree on this but might not agree on whether slavery should exist, or expand beyond the areas where it existed at a given time. There was lots of controversy over how to respond to the Mexican War, which was most heavily promoted by people who wanted to expand slavery. Southern Whigs might be tempted by the possibility of new slave lands but dislike the expansion of executive power which came with war, and northern Whigs were unhappy with both. They needed to find a way to unify against the war in a way which didn't break the party over slavery.

The second and third issues were immigration and temperance. In the case of temperance, a faction was in favor because, in addition to any issues with the problems of alcohol in itself, alcohol was associated with immigrants and Catholics, but Democrats could also seize on the temperance issue and therefore some Whigs were suspicious of adopting it as part of their platform. Whigs in one part of the country might take it up when Whigs elsewhere would reject it, or Democrats in the area would get it as their signature issue first (as far as state and local elections) and this could cause friction between different factions at a national level. Immigration was an even bigger problem - most Whigs were against it because most Democrats were for it and also partly out of sincere nativism on the part of many, who were at risk of defecting to a third party, the Native American Party. Some Whigs, notably William Seward, were in favor of immigration because they knew immigrants were a big voting bloc and felt they could get a chunk of those votes by compromising on a few issues such as Catholic school funding. Obviously this also caused friction.

Finally, the elected Whig presidents were, to some extent, all failures. William Henry Harrison died almost immediately after starting his term, and his vice president John Tyler had been a compromise candidate who didn't adhere strongly to Whig principles and therefore irritated a lot of Whigs once he revealed his true colours after succeeding Harrison. Zachary Taylor ran on the basis that he was a war hero and many Whigs thought he was their best chance of taking the White House in 1848, but he also turned out to be wobbly on certain key Whig issues and very wobbly on patronage (this was the age of the spoils system of openly giving government positions to party loyalists) and also died partway through his term. Also, the large chunk of land taken from Mexico needed to be settled in terms of whether slavery would be permitted there, and Taylor and his vice president/successor Millard Fillmore ended up taking a path, the Compromise of 1850, which irritated everyone -Northerners because it was too permissive of slavery, Southerners because only part of the new lands were opened for slavery, with California becoming a free state. Southerners, however, felt they came out slightly better from the deal, and Fillmore and a Southern faction insisted on not questioning the Compromise in the party platform for the 1852 presidential election. Everyone in the party was mad after the Democrats won the election and decided to bide their time until the 1856 election, thinking the Democrats surely would screw up somehow and the Whigs could make a come back, and in the meantime, a new organization called the Know Nothings was becoming popular, dedicated to resisting immigration and using oaths to binds its membership to secrecy and vote in a manner decided upon by the organization. Many Whigs decided to try and coopt it, thinking that with the secrecy thing, they could obscure the fact that they were setting the agenda in regards to how the membership should vote, and proceeded to run Fillmore for president in 1856, but the cooption was insufficiently effective and the slavery issue was a hair more pressing. This meant the Whigs had killed both the Whig party by trying to coopt the Know Nothings and to some extent the Know Nothings by means of the attempted cooption, and the Republican party ended up replacing them.