While it is unlikely that when a noble with the stature of someone like (Vlad III) Dracula graced a castle with his presence that they would bring anything less than a sizable following, in addition to whatever residents - skeleton crew if you will - already resided, lived and worked there, you do touch upon an interesting notion. That is that medieval castles, depending on the definition of castle as well as its location and function, could have very small garrisons, as little as six men as a skeleton crew to keep things "operational". Of course these castles would be of a different nature and function than the quasi-palaces a Wallachian prince would frequent with his following in the 15th century.

However, if we look at some different types of castles that had a pure military function, especially in the early and high middle ages (say, 500-1300 to make it convenient) they could be of a size and compactness that justified having very few men residing in them, especially in (relatively) peaceful times. But even in markedly less peaceful times, small castles and fortifications - such as fortified houses - were designed to be manned and defended with small number of men. It wasn't unheard of to have a smaller but well designed castle to be defended in a siege by about thirty men, surrounded by armies of a couple hundreds, or even a thousand men or more, such as in the month-long 1374 siege of the Gildenburg. This sounds heroic, but the attackers had only one way up to the castle and they preferred to wait things out and bombard the defenders into submission, ironically putting so much strain on two catapults that they broke down and they were forced to storm the castle with a siege tower eventually. By the end, nine out of thirty defenders had died.

On the other end of the spectrum were the massive castles that were more akin to full fledged military bases, such as Krak des Chevaliers. These could house hundreds or thousands of men, which fitted their function as a base of operations in deeply hostile territories, complete with a souterrain and indoor stable facilities.

To conclude, while in the context of Vlad III or the rulers and nobles often depicted in Gothic films and literature it is unlikely that they would be (near) solitary residents (who the heck is going to do your cooking, your laundry, the cleaning, the candles and the hearth?), it's feasible and accounted for in the sources that some purely military castles had a very small number of inhabitants, especially in peace time.