The question of which President is the best or worst seems a bit nonsensical to me...but you never know. I like to think that Stephen Colbert has competent editors and this sounds just a bit too direct to be solely humorous.
He was involved in scandals, but so were many Presidents.
So: is Harding really the worst? Anywhere close? Or is the question (all but) impossible to answer?
The segment in which Colbert makes the statement is available here, at about the 5:20 mark: https://youtu.be/5gcojKy6c7Q
This list from C-SPAN has credibility. It consistently ranks Harding a few steps from the bottom. This ranking is based on the collective wisdom of a number of historians who provide a cumulative point system. Ranking changes over time because of the addition of new ex-presidents (sitting presidents are not ranked). In addition ranking changes as perspectives change - this is particularly true of more recent incumbents who are particularly susceptible to changes in perspective.
Colbert isn't far from correct - and Harding is certainly the worst since the past 140+ years, so we have to tip our hat to Harding: he is clearly the agreed-upon worst president since Reconstruction - at least so far!