It was less to do with being sourced in morality, and more to do with common sense and the awkwardness of deviating from it. Hats (mostly for men) are meant to be worn as an out of doors addition to the ensemble. Hats are less common today, but are still meant for the same purpose. Whether it is a baseball cap worn to shade from the sun or a knit cap for cold winter. Think of it in a more modern sense, if it is very cold out and you put on a winter coat, scarf, hat, gloves, etc. it's expected that if you are inside of a place for more than a temporary moment, you remove these items. You may keep them on when buying fast food, but if you are inside of a persons home or spending a while somewhere you remove them (pending unusual temperatures inside, of course). If your friend came over and kept all of that extra clothing on while eating dinner or watching a movie on the couch you would think them either very odd or be concerned about your interior climate. Therefore, putting your host at such un-ease is considered rude. This concept is then made a hard rule, easier to say "remove your hat when indoors" than "remove your hat if you are expecting to be in a building for more than X minutes, will be sitting, or should be at ease in the location". Hats are only noticeable to us today because they are a less common item than something like gloves or coats. There are historical and religious exceptions to this being the main purpose as well, such as the large hats being worn by women in the early 1900s and the requests to remove them when seated in a theatre. But, again, that harkens back to being aware of those around you and how you may make them inconvenienced or uncomfortable.