While they wouldn't carry them everywhere, many would claim erroneously that swords were too heavy to carry around. This is not true, the average sword, even in the early medieval ages, almost never exceeded five pounds and was usually significantly less. A sword was only carried for two very specific circumstances - war and ceremony.

One would bring one's sword if the risk of violence was present, for obvious reasons. But while one wouldn't walk around the house or go about socialization with a sword strapped to your side, it was common practice to carry one's sword about for various ceremonies or public events. A sword, beyond a weapon, was a status symbol - it would be displayed at various events were one's status was something to be indicated. But these swords, while usually functional, were highly ornamental, and would not be the sort one would want to bring into battle.

So one would not carry a sword for any reason other than for safety, or to display status at public ceremonies or events - usually the larger sort were one would need to make sure one's higher status was visible.

And as far as I have read, there was no inherent stigma in bringing one's sword into another lord's castle - as generally speaking one had to travel to get to that castle. And one of the most common times to carry one's sword as when one had to travel, as even though banditry wasn't as common as many think during the middle ages 'better safe than sorry' as the saying goes. Though, of course, if asked to remove one's sword -especially when entering the manor-house of the lord (as the castle itself wasn't really the lord's house - castle's were military installments and as such weaponry was pretty inconspicuous) - it would then be rude not to comply without good reason.