Often I see TV shows and movies depicting castles as their primary residence. How accurate is this? Was it more true at certain points in history?
This varies immensly with time and place.
There were castles in Sweden that were the residence of the appointed tax collector, which usually was a nobleman (and sometimes a mercenary captain rewarded by the King with the position) - these type castles would see many different residents and garrisons over the years, and were frequently destroyed in uprisings.
Other castles were very lightly (or not at all) garrisoned and intended to be used in times of trouble. The royal castles built in Wales by King Edward of England usually had a minimal garrison. Conwy's, for example had a constable (commanding the garrison), 30 soldiers, 4 artisans and a chaplain. The Scottish clan castles were often of this type.
Other castles were the holdings of a specific family and were manors as well - they had land, cattle, business, serfs (or tenants) and perhaps even industry attached to them, and a caretaker or the owner himself in residence.
So yes, many of the more powerful and influential people of the era, nobility who owned a lot of land, lived in castles or fortified manor houses that were castle-like. However, not all castles were manors or even especially suitable for long-time residence but rather built as a short-time refuge in times of trouble.
In order to answer your question, we must divide castles into several groups:
http://images40.fotosik.pl/1704/8487b5248c31c55c.jpg
http://mw2.google.com/mw-panoramio/photos/medium/80556511.jpg
http://www.kogis.pl/gallery/medium/ph744tofertar6786.jpg
http://upload.wikimedia.org/wikipedia/commons/c/ce/Ireland_tower_house_near_quin_county_clare.jpg
Such castles were always lived in, and were designed to be so. The internal arrangements of such a castle would combine the qualities of a large mansion, a government building and a set of internal workshops, kitchens, cellars, storages, dungeons etc. These castles would often be joined archtecturally with a church or a cathedral, often with a roofed passage.
This "pallace-castles" were always lived in, and it was quite a prestige to be awarded one. Not all of those belonged to the monarch and were held by a nobleman; for example, one of the biggest castle-complexes in all history, the Marienburg/Malbork castle, which perfectly fits this description was a home to the Teutonic Order of monk-knights.
http://www.pomorskie.eu/res/kultura/zamki/malbork1_450.jpg
To sum up: Castles were occupied when it was profitable or prestigious to do so. The main problem with living in a castle, is that it was usually far less comfortable than living in a mansion, and that it was usually positioned away from where everyday life happened. Castles were built on hills or next to cliffs, on small islands, and in the forks of a river, while most cities, villages and manors were built on a plain, next to an easily accesible riverbank. This meant that unless the distance between the castle and the town/village was small, it would be a nuissance to dwell in a castle, and rural manors were prefered instead.
Another note: for the first several centuries of the medieval era, monarchs would rarely live in one place (castle or otherwise) but rether travel along the path linking the castles and manors of their retainers, to gevern things directly (so called rex ambulans). Historical sources claim that the arrvial of the monarch and his court was a great prestige for the local lord, but also that housing such a guest was incredibly expensive, often enough to bankrupt minor lords.
Sources:
Thompson, M. W.,- The Rise of the Castle
Guerquin B. - Castles of Poland
Kajzer L.- Lexicon of castles
Kajzer L.- Castles and society