Here is a previous r/AskHistorians thread with an answer that links to several similar answers.

This could really vary depending on the type of castle, the time period, and the resources an labour available. It is also important to note that castles we see today have largely been rebuilt, added onto, renovated, or damaged over the centuries. For example, Warwick Castle in the English midland began as a wooden motte-and-bailey construction out of wood shortly after the Norman Conquest in 1066. It was later converted to stone in the latter half of the 12th Century. This part of the castle is presently ruined as it fell into disuse. The curtain wall was expanded several times over the following century and several towers and an elaborate gatehouse were added in the 14th and 15th Centuries. Eventually all the original outbuildings inside the curtain wall were replaced with a 17th Century Manor House which was heavily refurbished in the 19th and 20th Centuries. Really, work on a castle never really ends, as it was imperative to keep them up-to-date on military technology.

I specialize in English and French history from 1066-about 1200. I'm afraid I don't know too much about castle-building in other regions and how it differed from England/France, so anyone can feel free to add on to represent other regions/time periods.

At the time of the Norman Conquest, stone castles were not commonly found in England. This is because England was relatively peaceful and centralised compared to France, where local Lords were in constant competition for land and wealth, causing a smaller-scale arms race in castle-building and siege warfare. One of the first things William the Conqueror did when landing in England in 1066 was put up a wooden motte-and-bailey structure at Hastings. Given that William landed in September and the Battle of Hastings was fought on 14 October, this castle was erected in a matter of weeks or even days. This type of castle, which served as the basis for many castles in England today, consisted of a man-made dirt hill with a defensible wooden structure on top. The local peasants would have been conscripted to do manual labour on building the mound. William used this as a base from which to harass the surrounding countryside. Original mottes on which wooden castles were built still exist in Warwick, Oxford, and Fotheringhay to name a few. (The Anglo-Saxon Chronicle and other sources actually disagree as to whether or not there was actually a motte or earthwork at Hastings, and considering that the castle has been rebuilt in stone, it's impossible to know for certain).

After his victory, William established several more castles surrounding London. R. Allen Brown states that between 1066 and his death in 1087, William established at least 36 castles of his own, and that his subordinates established far more. By the end of the 11th C. only a few of these were in stone, including the Tower of London's White Tower, which Brown states may not have been completed on William's death in 1087 but was certainly completed by 1100. According to the Anglo-Saxon Chronicle, building continued in 1097:

Many counties also that were confined to London by work, were grievously oppressed on account of the wall that they were building about the tower, and the bridge that was nearly all afloat, and the work of the king's hall that they were building at Westminster; and many men perished thereby.

Construction on the Tower would continue intermittently for the remainder of its use as a royal Residence until the Tudor period. There was once a royal palace on the site, which is no longer standing, in addition to the Keep and curtain walls, which are. It was heavily renovated by Richard I and Edward I and remained both a symbol of the monarchy and the most defensible royal base in the area in and around London.

Between William I's death and the middle of the 12th Century, there is a period where a lot of the original wooden motte-and-bailey castles were being converted to stone. This continued well into latter half of the 12th C as well. Unfortunately, I could not find many records of the actual costs or figures regarding labour for this period. Given that England was in a state of civil war between 1135 and 1154, I suspect that building works were sporadic in some places. Castle ownership was changing frequently during this time period as rival claimants Stephen and Matilda assigned conflicting lords as owners of the same lands, and re-appointed lords in response to changing alliances. It is also during this period that we know many new castles were raised. However, many of these were considered "adulterine" as they were raised without Royal consent and were demolished after Henry II took the throne in 1154.

Cont...