To a large degree, the Whig party rose and fell with the fortunes of one man - Henry Clay. In several elections between 1828 and 1846 Clay was thought to be the front-runner for the Presidency, but was thwarted (at first by Andrew Jackson but later by Clay's own Whig party). What Clay was much admired for during the peak of his career was his ability to forge compromise, first with the issue of Tariffs and then with the issue of Slavery. But the seeds of Clay's - and by extension the entire Whig party's - descent into irrelevance was the result of those compromises.
In the case of the compromise over tariffs, the tariffs were extended but gradually phased out. This deprived the Federal government of income that Clay and the Whig party intended to be used for large public works projects (building roads, canals, etc.). The great public projects that were part of Clay's vision of the American System were delayed for many decades. And the Northern manufactiring interests lost the protection that was an inducement for them to support the Whig party.
The compromise over slavery was unsatisfactory to both the growing abolitionist sentiments in the North and the slave-holding, States Rights South. The strengthened fugitive slave act created strong hostility in the North and accelerated the anti-slavery movement. Henry Clay tried to take the middle course - being a slave holder most of his career but late in life freeing his slaves. But in the anti-slave North, Clay's personal involvement in slavery was seen as being morally corrupt.
One of the great ironies of American political history was the election of William Henry Harrison. Harrison was chosen over Clay to be the party's nominee at a time when Clay thought for sure it was going to be his turn to win the Presidency after the hugely unpopular Van Buren presidency. Harrison, a military hero who was the Whig antidote to Jackson, was elected (in the original 'Log Cabin' campaign), but the truth was that none of the Whigs really knew for sure what Harrison's politics would really be. They just knew he was popular and electable. Harrison purposely kept his views close to his vest so as to not open himself up for attacks by his political opponents. Unfortunately Harrison died within weeks of being elected, and the Presidency fell to the Whig-selected Vice President John Tyler, who was even more of an unknown than Harrison. It turned out that Tyler was jealous of Clay and became uncooperative and actively hostile to Clay and the Whig party leadership. Eventually Tyler became allied with the Democratic leadership. While in office Tyler thwarted the Whig agenda (like re-chartering the US Bank) while presiding over the annexation of Texas as a slave state and preparing the stage for the war with Mexico. So the man the Whigs succeeded in getting into the Oval Office ended up damaging them severely.
After the Polk presidency, the Whigs got Zachary Taylor (who also died in office) and his successor Millard Fillmore. Both of these men continued in the weak, compromising vein that the Whigs had followed under the leadership of Clay. The deep divisions of the nation were not resolved, and it became clear that the middle ground approach of the Whig party would not be able to accomplish much of value going forward. In this time Henry Clay died; and really the Whig party died with Clay. It seems that Clay was the personification of the Whig party, and when Clay's political stances became old and largely irrelevant, the Whig party became irrelevant as well.