You should probably specify which culture and period you're talking about, because your premise certainly isn't true everywhere.

In the subcontinent, the Milky Way has long been an integral part of culture and popular imagination. In the Hindu religion, it's likened to a river in the sky, in fact it's the holy river Ganges, flowing through heaven, before it descends to earth on Mount Kailash, via Shiva's tresses. It's called "akashganga" which literally means the Ganges of the sky. Mythology is full of references to it.

For Muslims in the subcontinent, it's known as "kehkashan" in Urdu (originally from Arabic). It's celebrated for its beauty and is a fixture in Urdu poetry. In fact, it's popular enough that "Kehkashan" is a fairly common girls' name in the subcontinent.

I think you may be right that it's not mentioned as frequently as the sun or moon, but those are much more prominent objects. But it's definitely as popular as any star or constellation. Maybe not as popular as some prominent planets, mostly because planets move, which gives them a role in astrology.