I can only speak to the Anglo-Saxon and Norman fortifications, I’m sure other people on here are more well versed on continental castles. I think Norman castles are a great place to start considering it was such a landmark (both literally and figuratively) of the Normans.

Before the Norman takeover of Anglo-Saxon nobility in 1066, Anglo-Saxon defensive fortifications were built in the style of “burhs”. That is also the origin of the word borough, so if you’re English you can get a general idea of what they were. Burhs were fortified defensive communal towns where people were able live in safety against primarily Viking raids. [1] Alfred the Great commissioned these burhs in the 9th Century to ensure that there was a set of defensive fortifications connected by roads and patrols in Wessex to fend off enemy raids and generally facilitate Saxon trade. So, these burhs were not castles necessarily, but it gives you an idea of what was being constructed in Anglo-Saxon England before the castles we think of.

Norman castles, on the other hand, mostly began as timber fortifications in around 9th Century France. They were built as a feature of the feudal system (which did not make its way to England fully until the Normans), because as resources became scarce after the Carolingian empire’s decline, French nobles needed to secure their holdings with their own resources. In the early feudal system, knights and warriors would be housed within the lord’s wooden castle to protect them and create a military base of strength within the lord’s demesne. [2] Once stone began to be used, they would regularly start off as stone houses that had a tower constructed onto it. Stone towers became popular features in Norman castles and were sometimes built before the actual foundations of the keep. The Tower of London was one of the first Norman structures built in England.

Anglo-Norman castles were mostly built on the south coast at first because that was where most of the landed Norman nobility had their holdings, but these fortifications actually got a bit out of control. Building a castle would actually have the unintended consequence of spawning a village around it because peasants, tenants, and petty nobility liked the protection that having the knights and lord’s men close at hand provided them. The unintended consequence was that these castles started to be built by individual lords rather than central authority and the Norman nobility. [3] So, lords would build castles on their borders to have a consolidated position of power on the border of their lands, and consequently a bunch of villagers would prop up a town outside of it, as a nice target for an enemy raid.

Now that we’ve established why and where Norman castles were built, we can actually answer your question and go into the details of how they were built. Many castles were constructed in areas of unrest like on the borders of Scotland and Wales. The actual construction of these fortifications would be more perilous than in the heavily guarded areas of London. The Normans often used established architecture from the Romans and Saxons as a base for their castles, simply ameliorating the fortifications and walls with stone. These were usually improved in long stages over the course of several years, where the construction could be dropped and picked up again easily at a later time without worry if an attack were to occur. Stone castles (which would take much longer than earth and timber ones) actually weren’t all that popular until the 12th Century. When they became popularised, they were built in the ‘motte-and-bailey’ type. The simplest improvement you could make to a motte and bailey type earthen and timber-built castle was to strip away the timber bailey palisade and replace it with a stone wall. [4] The best thing about this strategy was that the masons that the lord employed to build and improve upon these fortifications could simply go inside the castle if there were any sort of threat. Presumably, there would have also been ample warning from scouts for the builders to run away or get inside. Most of these improvements and additions to Norman stone castles took little time as well, as lords had incentive to provide the masons and builders with all the resources required to get the job done efficiently and quickly. Castles were not only great defensive structures, they were status symbols as well. They represented the consolidation of Norman power and the strength of the Anglo-Norman cultural fusion that occurred in the subsequent decades.

TLDR: Norman castles were often built on the sites of Roman and Anglo-Saxon architecture, meaning the construction times would be reduced because they were not building from scratch. Most castles until the 12th century were built with timber, and laid the foundations to be improved by stone in sustained periods of improvement. Building could take anywhere from a year to twenty years, with motte and Bailey fortifications being built quickly as placeholders while under threat of attack until improvements were able to be made. Masons, builders, and architects could work on the castle presumably uninterrupted until scouts brought word of imminent danger. Castles built in dangerous border areas would have taken much longer to complete and were built in different styles to maximise efficiency and military security. So, it was probably common for builders to have to pause their construction projects on castles until a siege or raid was complete. But the infrastructure was such that they could pick up where they left off quite easily, which facilitated quick construction and partially explains why the Normans were able to build so many of them.

References:

[1] Gravett, Christopher. Norman Stone Castles: The British Isles 1066-1216. (Oxford: Osprey, 2003), 4.

[2] Gravett, Norman Stone Castles, 5.

[3] Gravett, Norman Stone Castles, 37.

[4] Gravett, Norman Stone Castles, 12.