Yes, James Madison was only 5"4 and Charles Pinckney was 5"9.
Obama (6'1") beat Romney (6'2"). Sources.
John Adams (5'7") beat Thomas Jefferson (6'2").