You might have to define who gets to be the common man:

Here's a copy of the Indian Removal Act. http://memory.loc.gov/cgi-bin/ampage?collId=llsl&fileName=004/llsl004.db&recNum=458 Summary of Slavery and its contribution to Jackson's wealth: http://www.thehermitage.com/mansion-grounds/farm/slavery

To some 'common men' and 'common women' he might not have been such a stellar fellow.

The Bank Panic of 1837 and the ensuing recession probably did not help the 'common man' either, and were in part [but certainly not wholly] driven by the rise of state banks [a result in part of the dissolution of the Bank of the US in 1832] and the state bank's habit of not holding appropriate reserves. [nice little article about it here http://www.library.hbs.edu/hc/crises/1837.html ]

TL;DR Jackson's terms might not have been that great a time for the 'common man' - depending on who they were.