I was just reading through some election stats on that bastion of reliability, Wikipedia. I noticed an interesting stat on this page. When sorted by voter turnout, we see 1824 way down at the bottom at just 26.9%, with the second lowest at a much higher (and more in line with every other year) percentage of 48.9%.
Even considering that 1824 was the first year turnout was recorded, it's still incredibly low - the following two elections exceeded 50% in turnout.
So what happened in 1824 that caused such a low turnout? I don't think we were involved in any wars, and I can't think of anything that was happening that year that could have affected the election so much. So what was different?
Without speaking about the exact political climate at the time, it was the first time ever no candidate ran as a Federalist.
Looking just a little bit into this, we see this source talk about why the party collapsed, which might have been why people were despondent for the election:
Tensions increased so much so that by 1814, some Federalists in New England threatened to secede from the United States to form their own country unless the American government immediately sought peace. With the signing of the Treaty of Ghent in 1814 and the end of the War of 1812, many Americans viewed the Federalists as traitors. The Federalist Party collapsed, leaving the Democratic-Republican Party as the only political party in the United States until the mid-1820s.
Since there was no one running as a Federalist, it might be inferred many didn't care who would become president. The Federalists (how ever many were left) didn't go vote because they didn't have a candidate. The Democratic-Republican party had five candidates running, so perhaps many people didn't mind who it was, as long as it wasn't one of those 'traitorous Federalists'.
Here's one last source from Brittannica that discusses how the King Caucus system failed and the election of 1824 was also the "first in which a large majority of electors were chosen by voters rather than by appointment by state legislatures".
The 1824 election was really the middle of a turning point. The number of states using popular election rather than state legislatures to select presidential electors rose from 50% in 1812 to 75% in 1824 (and 90% in 1828). Meanwhile the factionalisation and gradual decline of Jefferson's Democratic-Republican Party gave way to the rise of the Jacksonian Democratic Party in 1828, heralding the start of the Second Party System. The Democrats played an essential role in mobilising voters in 1828, including in areas where turnout had traditionally been low such as among German voters and in more rural counties. Political parties also helped increase the perceived relevance of the presidency. In 1824 the turnout for the presidential elections was lower than the average for congressional and gubernatorial elections between 1820 and 1825 in all but two states. The competition for the presidency, said Virgnia senator John Taylor, made "partisans zealous, and a great majority of the people indifferent".
Update: there's a good graph showing VEP turnout rates for both presidential and midterm elections here. As you can see midterm elections generally had higher turnouts until the 1830s, while the 1824 turnout was not exceptionally low for the pre-1828 era. The tiny turnout in 1820 was because Monroe ran essentially unopposed.