As with many customs in society, it is difficult to point out an exact time or place of origin for this one. Sadly I do not have a source to address this question directly, however I can draw on many primary and secondary sources of ancient to early modern clothing which may shed light on this topic when taken in context.

In context, we can infer the practical reasons why this custom originated and discuss why it became prevalent in Western civilization. Hats and other head adornments have had a wide range of functional, symbolic, and religious uses. We can discuss several applications of headgear over the millenia that led to them becoming symbols of status and why a tradition of removing these pieces in certain circumstances would arise. However we are going to focus on the medieval period onward, as this seems to be the earliest time we can comfortably discuss this custom.

Let us examine the symbolic/religious applications of headgear. Many ancient and medieval civilizations believed their leaders to be divine or atleast representatives of their gods/religion. These beliefs would culminate in the rise of the Divine Right of Kings on Europe. See this paper about James I to read more about the Divine Right of Kings. In this example, we see the belief in James I that he was the head of the church and had been given divine right to rule the English people. The ultimate symbol of this power would have been the crown.

A crown, while not a hat, is an example of extreme religious as well as secular status expressed in something worn on the head. As this is an ultimate expression of status for a king, so would lesser members of society wear headgear or functional pieces that come to represent their role such as priests and bishops or knights and soldiers. Likewise in later periods we can point to the elaborate military hats like shakos, bearskins, tricorns, bicorns, etc which often were more an expression of symbol or status than of function.

The association of elaborate headgear like crowns, helmets, military headdress, and priestly coverings with high status would naturally translate to the more functional hats worn by everyday farmers, merchants, and the like seeing their common headdresses as symbols of their occupation and lower status in society.

In short, by the medieval period, hats were associated with one's place in society and served as a sort of symbol of who one was.

From this we can start to surmise why a tradition of doffing a hat might arise. It likely arose as a custom of functionality which became symbolic for less functional headgear. For example, if a nobleman serving as a knight in medieval Europe were to approach the king shortly before or after a battle, or during a tournament, it would only be a natural sign of respect to remove his helmet to show his face and his intentions while speaking to his ruler. That same knight may remove his helmet for religious observations, showing his respect. Inversely, if he were raiding a village in a far away land, he may wander into an abbey or cathedral with ill intentions and keep the helmet on. Likewise, he may keep his helmet on when addressing a serf on his land or an archer walking by in a military camp. Also, wearing a hat or other headgear may impair the vision or space of those around you in confined spaces. Often with the style of architecture in medieval Europe, the door frame or ceiling height may require you to take off your hat or headgear for ease of movement as well.

When we relate this functional sign of respect to the idea of hats a symbols of status, we can see why one might start removing their hat in the presence of their betters.

But how could removing a hat in front of your king or noble become a tradition of removing your hat indoors? Well, as a common member of society, you are most likely to run into your superiors indoors in their home or castle. Or as a noble in the kings court. As we have covered thus far, your hat was your status symbol. Naturally removing this becomes a symbol of deference in the company of those above you, a sign of respect.

It is easy then to imagine how then this sign of respect could be paid to peers as well when when entering their home.

As hats become more elaborate through the medieval period and beyond, it becomes a matter of tradition as well as function to remove it when indoors.

Military Headwear would continue to grow larger and more elaborate until the shakos and bearskins of the napoleonic wars. It would become a matter of necessity to remove these tall hats indoors. Around this period and a little after the top hat becomes wildly popular and the same logic above can be applied here.

In conclusion, we can see how hats became as much a symbol as a functional item and how evolving societal norms and functional use of headgear led to a custom of removing one's hat when indoors.

Post Script:

There is alot more to say honestly, but this reply is already delving into stream of consciousness. Maybe in the comments we can touch on the idea of the hat as an outer garment playing a role in its removal indoors.