I'm surprised by how widespread this superstition is. I've for example heard that Romanians traditionally believe it attracts death, according to Turkish and Japanese traditions you invite devils or demons into the house, while Indians believe it attracts (supernatural?) thieves.
This is one of those folk superstitions I just can't wrap my head around. I'm wondering if it an excuse to scare annoying kids or if there is something else going on here. Are the superstitions connected? What could be their common source, if any?
The most common formulation of the superstition in Japan is not about “whistling in the house” but whistling at nighttime.
Why whistling? There is actually a book about this, with the lengthy title Japanese People’s Customs Which are Terrifying to Know: The True Reason for ‘Not Whistling at Night’ (千葉公慈著『知れば恐ろしい日本人の風習 : 「夜に口笛を吹いてはならない」の本当の理由とは』 河出書房新社, 2012). According to this book, in ancient times whistling was called usobuki in Japanese meaning a high-pitched sound like the cry of a bullfinch. Such sounds were associated with rituals calling upon gods and spirits. (In older Japanese films, this eerie sound is evoked by the shakuhachi flute.)
The theory goes that the proper setting for calling spirits is in an organized ritual arranged by specialists, whereas if you call spirits by yourself at night, bad ones might come and visit misfortune upon you. For this reason whistling is seen as a frivolous and thoughtless activity and some superstitions are connected to it.
This logic is reasonable to me in the Japanese context because it matches other ancient beliefs about evening. Of course, in an era before artificial lighting, people did not go out at night; it was therefore said in texts like the Nihon Shoki that the daytime belonged to humans and the night belonged to spirits. The transitional time between day and night, dusk, was called tasogare which literally means “Who’s that?” and it was thought that as the light of day faded, spirits emerging at this time had an opportunity to bewilder people and spirit them away to other realms. This is why the first scenes of the anime Spirited Away take place at dusk. But it is natural that superstitions would also develop about the type of activities permitted at night. I can imagine that suddenly hearing a shakuhachi-like sound at night might be scary to ancient Japanese people.
I do not know why you heard this superstition as “whistling in the house”. Presumably whistling in the house was not such a big deal as long as it was during the daytime. There are many regional variants of this superstition in Japan, such as “if you whistle at night snakes will appear” (snakes were seen as powerful and dangerous spirit messengers), or “do not whistle onboard a boat.” Perhaps one of them mentions houses. I am not an expert in Turkish or Indian culture but both countries are big places and I suspect that there is a similar generalization being made from a local superstition to a nationwide phenomenon.