I'm sure examples of casual clothing abound throughout history, but my favorite is that of Thomas Jefferson. When Senator William Plumer arrived to meet Jefferson at the President's House in Washington, D.C. on 1802, the president was "dressed, or rather undressed, with an old brown coat, red waist-coat, old corduroy small clothes, much soiled, wooled hose and slippers without heels." Imagine the uproar in contemporary media if a president conducted business dressed this way.
Source: