This entirely depends on the city, castle and noble in question. It should be noted primarily that throughout the Medieval period, the vast majority of the population lived in farming villages in a rural setting, and thus, if the lord lived nearby, it would also be in this rural setting. Indeed, the majority of castles in England were in rural settings. That doesn't mean they were isolated; I wrote an answer here about how, in frontier regions in particular, castles encouraged the accretion of settlement and farmland.

Urban castles were less common; indeed not all towns had a castle, although they were more common in cities. Cities in the period are somewhat of an organisational curiousity: governed by charters, most were overseen by a Reeve or Sheriff, but could be ruled often by a council of burghers, aldermen, guild members, bishops or other leading figures in the civil and commercial life of the city. Castles in this context could be royal centres rather than the residence of a specific lord, as of course in many settings there wasn't a specific lord who ruled a city. The most famous example is probably the Tower of London which, while guarding the city, was also a centre of royal control. Bristol Castle, is another example: a royal castle which not only acted as part of the city's defences, but also was a royal mint, and effectively also stood to safeguard the city's independence from the encroaching interests of the nobility from Somerset and Gloucestershire.