This varies massively depending on the castle in question but, broadly yes. Roman ruins that are open for display will often be mostly 'authentic' apart from reconstructions, if only because they are usually buried and have to be excavated. A couple of good examples would be Vindolanda on Hadrian's Wall or the Roman Palace at Fishbourne. For a genuine Roman stone that is still standing, one of the mile markers on Hadrian's Wall is still upright. However it's inscription is missing after over a millennium of being used a scratching post by cows.

To give a couple of examples of castles: The Tower of London has been in more or less continuous use since it was built in the 11th century. Obviously there's been some wear and tear over the years so some repairs and replacements have been made. As well as additions by various rulers.

For a castle I know more about personally, Pevensey Castle has an extremely long history with long periods of disuse before being reoccupied. The site was originally fortified by the Romans in the 3rd century, probably to help stop raiding Saxons. This eventually fell into disuse and was left until being reoccupied by the Normans in the 11th century. Apparently at least some of the Roman walls were still standing at this point as they built their structure inside them. Pevensey castle was then built up and saw a variety of owners and conflicts for the next 400 years or so before, again, falling into disuse and disrepair. It was nearly torn down on at least two occasions, once by Royal decree and again when the site was sold to a builder who wanted to quarry the stone. It would lie in ruin until World War 2 when Pevensey was noted as a potential landing site for a German invasion. So, after nearly 500 years out of service, Pevensey Castle was once again occupied by the British Army. They patched some of the holes in the walls, put down wooden flooring and built a blockhouse. After the war most of the military adjustments were removed, although they left the machine gun nests as evidence of the castle's most recent use.

I'm afraid I know far less about dry stone walling. The technique is ancient, going back to the Iron Age or earlier with sites like Skara Brae. But as for when a particular Welsh wall was built? You'd probably have to ask the farmer.