Have they all pretty much retired and gone into philanthropic work after leaving office?
Or have any of them ever gone back to being a senator, governor or held any public office?
I don't know if this counts as politics, but William Howard Taft was appointed as Chief Justice of the Supreme Court by President Harding, an honor that Taft prized more than the presidency.
The most notable would be John Quincy Adams, he decided to go to Congress after his term. And in the end served over 17 years as a Representative from his home state of Massachusetts. While not quite the leading voice that the "Great Triumvirate" of John C Calhoun his former Vice President, his fellow Bay State Daniel Webster, and his one time SECSTATE Henry Clay, Quincy Adams remained in some ways an anomaly of the still active President and in some ways the elder statesman.
One the topic of Slavery he also grew to become a staunch and outspoken opponent of the peculiar institution. Along with opposing other related measures such as the annexation of Texas and the Mexican War. He also was key in protecting what would become the Smithsonian Institute today, and his authorship of a compromise tariff was key in deescalating the Nullification crisis under Jackson's presidency.
But one of the most famous of his post presidential episodes was when he was asked to join the defense in the Amistad Case. A case before the Supreme Court about whether the human cargo of a Spanish slave ship who had rose up and eventually landed in the US should be considered as free men or shipped back into Spanish hands. Despite opposition from Spain and pressure from the Van Buren Administration the USSC found them to be lawfully free, based in part upon Adams; arguments and the US having outlawed participation in the international slave trade.
Grover Cleveland was elected president a second time in 1892, after losing his re-election bid in 1888. The only president to serve non-consecutive terms.
Martin Van Buren left office in 1840. In 1844 he attempted to get the Democratic nomination for president, but his opposition to the annexation of Texas made him too unpopular with southern Democrats.
After this he became increasingly active in anti-slavery politics. In 1848 he ran for president again as the Free Soil candidate. While he did not receive any electoral votes, he was successful in drawing votes away from the pro-slavery (or at least slavery-accepting) Democratic candidate.
He spent the next decade leading the small anti-slavery faction of the Democratic party. He publicly spoke out against the south's attempts to secede, but died soon after the civil war began.
John Tyler returned to his native Virginia after leaving office in 1844. An unpopular president, he was mockingly appointed "overseer of roads" by his community, and took the job far too seriously. He also led a militia unit that was raised to protect slave revolts and abolitionist raids.
Later on, he helped lead the secession process and was a strong proponent of the Confederacy. He was elected to the Confederate house of representatives, but died before they had a chance to do anything. His death was essentially ignored by the north, but he received a CSA state funeral. He was buried with a Confederate flag and was lionized as a "founding father" of the Confederacy for the next few years.
Andrew Johnson won a senate seat in Tennessee after his presidency. Awkwardly enough, many of the senators he served with had voted to impeach him. He didn't do very much as senator, as he died shortly after election.
Besides for holding elected office, former presidents tend to have a lot of political influence and often campaign or endorse candidates.