Let's start with the wearing of "robes" by Western men. We can likely date the appearance of wrapping gowns to the mid-17th century. They show up in Holland around the time the Dutch are beginning contact with the Japanese and are definitely originating in Kimonos. We also have the Banyan, a more closely fitted robe style coming from India and named after the Hindu word for "trader". It also appears in Europe in the 17th century. These are worn instead of a mans coat for leisure wear. It could be before getting fully dressed in the morning, hanging around the house, at work with no formal appointments, or even places like the coffee house or tavern. It's acceptable in public, but it's very casual (also very exotic and worldly). Since they can be made very light weight for hot weather or made quilted for colder days they're perfect for weather extremes.

You can watch through the 19th century as something that very closely resembles a Banyan becomes known as a dressing gown or lounging robe. While still associated as "undress" to be worn casually about the house, as their name implies, they aren't yet associated with bathing in the way they are today. Obviously, they would serve well during the in-between of bathing and dressing, but submersion in a bath tub as we think of it isn't universally common yet. American's are ahead of the European curve in installing running hot water and bath tubs or showers because they're building mostly new structures rather than retrofitting old ones. By the 1930s, 89% of New York City's apartments had baths or showers compared to the 3% in the early 1890s. For that reason you can find the evidence of "bath robes" for men by the 1910s. Notice how the one using the term bathrobe also calls it a lounging robe. That duality continues through to today, where you'll find advertisements for men's robes usually pairing them with pajamas as casual undress.

I would say it's not that the robe has a link to bathing, so much as being dressed in a way not suitable for being in public. "Dressing gown" is the term used in the UK (as I understand it), rather than bathrobe, and even in the US people do wear a robe over pajamas if it's cool enough.

The wrapping gown or robe entered the Western consciousness in the late 17th century as an import from the East, made of expensive materials and available only to the wealthy. Owning one - or, more importantly, being painted in one - was not just a sign of affluence, but also intellectual pursuits. Benjamin Rush wrote in the late 18th century:

Loose dresses contribute to the easy and vigorous exercise of the faculties of the mind. This remark is so obvious, and so generally known, that we find studious men are always painted in gowns, when they are seated in their libraries. . . . It is from the habits of mental ease and vigour which this careless form of dress creates, that learned men have often become contemptible for their slovenly appearance, when they mix with the world.

Wrapping/loose/morning robes trickled down into very informal dress early morning/late evening for most levels of society by the 19th century, and have stayed there ever since.

John Banckes, painted by Sir Godfrey Kneller in his robe in 1676

Sarah, Duchess of Marlborough, painted by Charles Jervas in hers around 1700