Part of the issue here is that you're looking at the formal/informal divide from a very modern viewpoint. As I explained in this answer, the twentieth century saw sportswear and "sloppy" student clothing become unremarkable everyday dress; as a result, anybody born after 1930 or so has a mindset on the subject that is fundamentally different from that of earlier generations. During the nineteenth century and earlier, people of means - that is, the relatively prosperous middle class and those above that - would spend very little time in clothes that we'd consider "casual". Everyday dress was informal because it wasn't ornately/expensively decorated or made out of the costliest fabrics or made in a particular court style, not because it was loose and suitable for sprawling on a sofa. As a result, we can conjecture that casual dress was made in a very similar manner to those we see in fine portraiture, just with fewer jewels and perhaps in more ordinary fabrics. That being said, we do see images of people not in formal dress, such as the portraits drawn by Holbein of the More family (like this one of Meg Giggs), or portraits of merchants from the Netherlands.

The kimono-shaped dressing gown (seen here on John Banckes in 1676) was not worn in Europe until the late seventeenth century, when it was imported from India, despite the fact that it seems like a very obvious thing to make; it could be used in portraiture (along with a cap in place of a wig, once wigs were in fashion) to show that the wearer was an intellectual with the means to a) afford a silk dressing gown in the Indian style and b) sit around in it rather than doing manual labor of some kind, while either reading/writing philosophy or doing high-level merchant tasks. Women were also depicted in these, or in imaginary dresses that gave the impression of a crossed-over front (see here on Sarah Churchill, Duchess of Marlborough ca. 1695), for similar reasons, minus the merchant tasks. Again, before this time period, we have no evidence of this kind of thing existing. The closest we come is to the "jumps" worn by noblewomen in portraiture that is either in the last few years of what can be considered Tudor - the very end of Elizabeth I's reign - or fully in the Stuart era. "Jumps" during this time refers to a lightweight, heavily embroidered jacket worn by elite or possibly also affluent women in extremely informal situations at home. You can see one in this portrait of Lady Arabella Stuart, or a later one on this unidentified woman. Despite the fact that these were informal, the embroidery could be quite elaborate and even involve gold and silver thread, lace, and sequins. (You may be interested in reading more about one such jacket that appeared in a portrait of Margaret Laton and actually still exists today. The linked blog post also has many macro shots of the embroidery and lace.)

As far as pajamas go, people didn't wear that much to bed. It's generally accepted that people wore either their shirt or shift/smock (see this previous post of mine on these garments) or a version of these made specifically for sleeping in. It's hard to say what definitely made a shirt or smock for sleeping rather than day wear - we can be confident that a very long man's shirt would be a nightshirt, for instance, because it wouldn't be able to be tucked into his hose, but beyond that ... We also know that nightcaps were worn. Andrew Boorde's 1547 First Boke of the Introduction of Knowledge recommended that they be made of scarlet, a red wool, for the warming properties of the wool and the color itself; Henry VIII had red wool nightcaps, as well as red satin ones that were likely worn during the day, despite their name, in informal situations.