In the most recent episode of GOT, there is a character who wears heavy plate armor in the middle of an almost impenetrable castle. Would knights or lords do this at all? Armor is heavy and took assistance in many cases to put on and take off, making it seem unlikely that people would wear it unless they were going to war or traveling.
I just wanted to address your point about the armor being heavy and unwieldy. In point of fact, it was actually relatively light, with a suit of plate weight about 60lbs (not talking here about the tournament armor which is a different beast entirely).
A suit of mail would weigh about 50lbs. By comparison, modern infantry can often carry 50lbs of gear into battle.
Here are some videos from the Metropolitan Museum of Art to help explain about the weight of armor.
How to Mount a Horse in Armor and Other Chivalric Problems (lecture given by Dirk H. Breiding the Assistant Curator of the Department of Arms and Armor)
A Visit To the Armor Galleries filmed by the Met in 1924. Has some of the same sorts of questions about weight and flexibility of armor and even includes someone from the museum staff trying on armor from the collection to see how it fits! The demonstration includes a seesaw with a small child and a fully armored knight in Central Park.
I know this isn't a sourced answer to the question, but more of a correction to the question itself (so, I'm sorry if it doesn't follow the sub's rules - feel free to destroy it) -
In Game of Thrones, people don't usually hang out in full armor. The guy you're likely referring to from last night is someone who's known for his armor - that's why he was wearing it inside, at a nonviolent meeting.
Please see /u/smileyman's comment for the details of the weight.
In reference to the show, the character you are referring to is Bronze Yohn Royce. He is famous for wearing bronze armour with strange runes on it, the show got this wrong but it was some kind of reference.
As for other people, it depends. Characters such as Brienne and The Hound are always in armour because they are travelling and the Seven Kingdoms are still in a state of unrest. There are bands of criminals and vigilantes roaming around. Other characters in arrmour are the Kingsguard, who are pretty much guarding the king at all times.
Ser Jorah actually wears wool and mail, sometimes even plate, in Essos. He is famous for this in the book. Essos has a hotter and drier climate than most of Westeros so that's why people there generally don't wear heavy plate.
Note that sometimes GRRM's world is not as historically accurate as people think.
I feel the need to add that the character in question is Bronze Yohn Royce, who explicitly is stated to wear his bronze cuirass at all times, as he believes it magically protects him from harm.