At least in the 17th and 18 centuries, there were notable differences between what men wore in public and in private, for which we have pictorial evidence since some were painted in "undress" as a sign of intellectualism. You can read more about homewear of the period in some earlier threads, like this one by u/mimicofmodes focusing on turbans, and this one by u/colevintage and u/chocolatepot focusing on banyans/wrapping gowns.

As for examples, you can compare Alexander Pope in daywear and undress, and here where he is not wearing any head-covering at all. Johann Joachim Winckelmann seems to have preferred portraits in undress, as seen by these three paintings of him. For some other examples, here are Diderot, painter Jonathan Richardson, and a British nobleman