It's a well worn trope in English History that the Normans brought Castles to England, that Castles were a vital part of the Norman Strategy for keeping control of their new conquest etc. And I'm aware of similar notions about the Crusader States/Crusading Orders like the Teutonic Knights etc.
But who actually designed and built them? I doubt the new Norman Nobility and the Crusaders bent their own backs. Did they import Architects and Stonemasons? (I'm aware the first Norman Castles were wooden.) Were their famous Castle-Designing lineages? Would these architects etc. By minor Nobility? Burghers?
The design of a stone castle was entrusted to the Master Mason, who combined jobs that we would now consider to be separate - Architect, Works Supervisor, Project Manager, Quality Control, Procurement, Human Resources. The Master Mason is different from a master mason - i.e. a qualified mason who has been accepted as capable of training apprentices. He was the man responsible for managing the entire process, doing the setting out, fixing the wages or piece-rate payments, etc.
The great Master Mason we know of was James of St George, who was brought over from Savoy (Alps) by Edward I and designed/built the four huge castles that cemented Edward's control of his newly conquered territory in Wales: Harlech, Conwy, Beaumaris, and Caernarfon. These were designed according to the most sophisticated fortification principles of the time, with concentric walls and multiple arrow slits facing in several directions. Despite being a renowned builder, he was not a member of the nobility and received his livelihood as salary payment (24 pence per day) or later as a pension from the king. The financial records that survive suggest that he brought some other Savoy masons with him. Masons were accustomed to spending their lives on the move as they had to travel to wherever the building work was, unlike carpenters and blacksmiths who often had enough general work in their area to allow them to settle down.
In terms of building the castles, this was done by large teams of craftsmen including masons, carpenters, labourers, blacksmiths, glaziers, etc. In the 13th century the king's officers could conscript these craftsmen and order them to travel halfway across the country to work on royal projects, on pain of imprisonment. For the four great castles in north Wales, which were built simultaneously and at breakneck speed, this required thousands of people and even at the rather basic pay of three or four pence a day it almost bankrupted the king.
There is a good article on this by L.R. Shelby, "The Role of the Master Mason in Medieval English Building" (Speculum, Vol. 39 No. 3 July 1964) if you want to read more.
Without getting too clinical in my answer, once the Norman’s had completed the Conquest, they imported their own builders and architects from across the channel to help oversee castle construction.
Initially, they were ‘motte and bailey’ castles, which were wooden and earthen.
It wasn’t until Plantagenet rule (specifically Edward I) that castles were routinely built out of stone. Edward I had a reputation for being quite brutal in his dealing with the Scottish and the Welsh - he put up stone castles to solidify his control over Northern England, parts of Wales and the Marches. Stone lasted longer than wood, in fact, many castles of the era still survive today in very good shape.
In answer to who built them; typically, the ‘builder’ is credited as the one who ordered its construction. Caerphilly Castle (in Wales) was built by Gilbert ‘the Red’ De Clare, the Lord of Glamorgan at the time. It was his home and a strategic chokepoint in the English’s defence/control of the Welsh population.
Many of the architects/designers of the era are simply lost to history, but those that do remain are fairly easy to find with a quick Google search.
The actual workers (cutting the stone/assembling it with mortar etc) were skilled stonemasons and were assisted with local labourers just like today. These stonemasons came from where ever the castle builder could get them from. Payment was offered in almost every case, and so whoever wanted the money did the work.
It’s important to note that castles have evolved over time, too. Windsor Castle’s architecture is far more recent than Caerphilly Castle’s architecture, for example. That’s because styles become updated, things get damaged/replaced etc. So a typical castle will have many builders/architects throughout its life, which can sometimes make it difficult to determine who may have built what.
Wales still has more castles per area than any other country on Earth if I’m not mistaken. This was because of the uprisings there and several English attempts to subjugate the native Welsh and overthrow the Tanist (Prince) ‘system’ they had there.
There's always more that can be said, but I thought you might also like this recent post from /u/CoeurdeLionne.
How long would it take to build a castle in Medieval England?