There were equivalents, although not quite as comfortable as track pants. This answer is going to focus on Western European/American women's clothing for the middle and upper class, because that's my area.
First, although fashions varied widely in silhouette, complexity and level of construction throughout the century, one concept (for those who could afford it) remained the same: the concept of "morning clothes". These were more informal, inexpensive and generally less fussy; they were a bridge between what you wore in bed and what you wore in public. You'd wear a morning dress to eat breakfast (unless you did that in bed), write letters, discuss domestic matters (like the day's menu, staffing, replacing worn linens, etc.) with the housekeeper, balance your accounts, etc. Later in the day, you'd switch to a more structured afternoon dress to go shopping, pay calls, etc.
Overall, morning clothes were lightweight, they were often light-colored or white, and even if structured, they were comparatively simple and necklines were more modest. Also - although this depended on personal taste - they could be extremely loose. Materials were mostly simple, breathable and light - linen and cotton, especially muslin, were favorites. By the way, if you've ever gone to the fabric store and touched what we now call muslin, it's not the same thing. Our muslin and all our quilting cotton are extremely stiff and thick in comparison. Muslin used to be almost gauzy - extremely thin and delicate. It was often printed with figured flowers, stripes, dots or checks, to give a bit of fun variety. Winter morning dresses definitely incorporated wool, though.
Afternoon dresses, in comparison, felt more like what we would feel if we were putting on a suit to go to work. They were more structured and decorated, more restrictive and darker in color, were made of heavier fabrics like silk and wool, and had more detailing. Of course, these weren't hard and fast rules - there are many gorgeous examples of white muslin afternoon dresses - but in general, you wore the less expensive, more comfy stuff to putter around the house and your fancier, fitted, pretty things to go out in public.
Another thing you could wear - if you were feeling especially lazy, sick, tired or pregnant - was a wrapper or a peignoir. These fluctuated in size/silhouette along with the current fashions, but the theory was the same - it's basically a dressing gown. It's really a robe pretending to be a dress. No corsets necessary. They could be fitted or entirely loose.
Now, on the subject of corsets: we're used to the idea of metal boning, and to those suuuuuper uncomfortable-looking corsets we see all over RenFaires that turn every woman's chest into a shelf and make it practically impossible to do anything, all while digging painfully into flesh and making the wearer extremely sweaty. In reality, Victorian corsets were much less restrictive than we like to think, and they were also much lighter. This is an amazing piece on antique corsets and whalebone. By the middle of the 19th century, there were actually machines that could weave boning directly into the fabric, meaning they could churn out extremely light single-layer corsets. They also didn't use much metal, saving that for the busk in the front. Whalebone, or baleen, is an extremely light material that has two fabulous properties for corsetmaking: it can be molded with heat and steam, and when it's been soaked in water, it can be sewn through. This meant that bones stayed in place, instead of working their way out of tearing fabric and stabbing the wearer, and it also meant that, thanks to body heat and perspiration, the corset conformed to the wearer's body over time.
Now, when corsets were manufactured in bulk by factories, they were generally molded into shape around hot metal dress forms, meaning that the whalebone would already be curved at the time of purchase, and a well-fitted corset gently curved over the curves of the bust like a bra. The seams and coutil fabric would keep the general silhouette intact; the whalebone would gently shape itself around the wearer's body.
This was actually especially advantageous for women who had more body fat - fat essentially fills out whatever shape the corset created, meaning it was quite easy for them to achieve the exact curvaceous hourglass silhouette we see in those old advertisements. r/CatsAreLiquid is an oddly accurate illustration of how this worked.
The final thing I want to address is tightlacing. We as a culture have a misconception of Victorian corsetry - as a matter of fact, very few people laced themselves painfully tiny. Working women especially didn't bother- the corset was there to provide structure and a silhouette, like a bra. Fantail lacing actually allows a person to put on and tighten their own corset, which meant it was an easy item to wear. A few special people did take pride in yanking their waists down below 19", and some even sent their daughters to schools that advertised a 10" waist reduction in two years, but overall, that wasn't the norm for the same reason that not every woman today wears a Kardashian waist trainer - it was uncomfortable, inconvenient and they didn't want to.
Essentially, even if a woman did wear her corset when relaxing at home, it was more like a long-line bra than a heavy, unforgiving, multi-layered brocade RenFaire contraption.
As for men - here I know less, but they also had morning clothes and dressing gowns. For the first 20 years of the century, however, you did not let yourself be seen with your shirt neck open; that was akin to walking around with your underwear hanging out. You could certainly choose to forego the heavily starched, 12" wide, intricately folded neckcloth and high, stiff shirtpoints of the dandy or Corinthian, and you could shock your conservative elders and show up wearing a Belcher neckcloth (Belcher was a famous boxer who wore a red bandana as a cravat), but you didn't wear nothing at all.
Taking your coat off and walking around in your shirtsleeves is probably the equivalent you're looking for. You'd never go out in public like that, but in the privacy of your own rooms, you could take off your coat.
Some men's fashions were entirely a matter of choice - e.g. corsets, coats fitted so tightly you had to have help putting them on, skintight pantaloons ("Inexpressibles") - so forgoing them for the sake of comfort would be completely normal, since most people didn't wear things like that anyway.
Fascinating - thanks for the wealth of info!!
You've had a great answer focusing mostly on women's dress already; I just wanted to note that I have a previous answer on men's clothing here: What did leisure wear look like for men in England in the 1800s?
I also want to note the fashion of the teagown. This is a style of dress that arose in the late 1870s as an option between a wrapper and a gown, either developed by or rapidly adopted by the Aesthetic and Rational Dress Movements (two closely linked groups, both interested in loosely-fitted and anti-fashionable clothing; the Aesthetes for reasons of what they considered beautiful and the latter for reasons of health). The teagown was intended to be something that a woman could wear in her own home for tea, either when by herself or when she was hosting other women, and be relaxed and comfortable in without a corset.
A hostess on her reception day may wear as plain or as elegant a gown as she pleases. She has a perfect right to make a lovely picture of herself against her own favorite background and a novel sort of toilet may fill up the deficit of any real interest in her, since an artistic combination of velvet and old lace or an aesthetic tea gown gives a pleasant impression for her guests to carry away.
If a breakfast gown expresses a woman's character and pursuits, a tea-gown may be said to show her tastes and caprices. It is of course an article de luxe in a wardrobe. It suggests boudoirs. One cannot imagine a woman doing anything heroic or strenuous in a tea-gown ...
("Women and Gowns", Lippincott's Monthly Magazine, December 1883)
Unlike a wrapper (likely what's being referred to as a "breakfast gown" in the above excerpt), the teagown was more than a plain lining with a layer of cotton or wool over it: it was beautiful, high-end and possibly even couture. In theory, they were worn without a corset and were completely loose, but in reality there was a wide range of possibilities, with some having boned linings that may have been used as light corseting or on top of a corset - most formal day and evening bodices were made with boning in order to keep them from wrinkling over a fully boned corset. Regardless of how they were worn, they were certainly intended to look uncorseted and unrestrictive, often with hanging panels in front or in back, little shaping, and/or or loose jacket effects (that is, made to look as though there were a loose jacket over the gown, when in reality the parts that look like a jacket were fully part of the gown):
Teagown from Liberty of London, ca. 1888
Teagown from Maison Worth, 1894
Sometimes, these were seen as appropriate for dinner at home, and semi-formal gowns were also designed with some influence from teagowns.
They continued to be used by wealthy women into the twentieth century, but 1909 saw the rise of the looser and more "natural" silhouette in fashion, which made the distinction between normal dress and teagowns much murkier and more subtle.