The reason the title says "house/castle" is because I'm primarily asking about houses, although I'm assuming that castles have a more well-documented history.
Out of all houses with fully documented history, which house is known for being the oldest house -- that's still both unaltered and inhabited?
I think you need to give a bit of latitude for the "unaltered" part.
Very few dwellings, if any, would be continuously occupied without at least interior modifications from time to time.