That depends on when and especially where you're talking about. It's a vague question so it's hard to give more than a vague answer. In general building customs were pretty rational, on the basis of the 3 Ms: Materials - What resources were available in terms of materials? Methods - What methods are known to exploit the material? Lots of god clay means nothing if you don't know how to make bricks. Means - How much money/resources do you have to spend on construction? It should be no surprise that there's more castles in wealthier and more densely populated areas.

To build with mortar and stone, you need to know how to make lime mortar which isn't entirely trivial. To take my home Baltic region or instance (and concentrate on the Middle Ages), it only reached Scandinavia in the 11th or early 12th centuries and even later in Finland and the Baltic states. The few stone structures prior to that are drystone (without mortar) and that type of construction requires you to use smaller stones, on average, as they need to fit together better to ensure a stable structure, when you don't have mortar filling out the gaps. Even after the method became known, places like Iceland didn't have the means even once they had the method; they couldn't produce mortar because there's no chalk or limestone there. They developed a tradition of turf houses where much of the walls are drystone with dirt on the outside to insulate it. As they didn't have mortar locally and had little wood, it was a sensible technique. Meanwhile in Norway, Sweden and Finland log houses dominated as wood was plentiful and it was easier and cheaper to build that way. Stone or brick was reserved for high status structures like castles and churches. (so that's materials and means, respectively) Denmark and large parts of Germany, France and England had wood, but not so much to be able to build log houses. Nor did they have much straight-grown Scots Pines or Norway Spruces anyway. But they did have very strong hardwoods like oak and beech, from which they could build half-timbered (or timber-framed) houses. Whereas in lots of Mediterranean Europe they didn't have even that, and stone houses dominated the traditional house-building techniques. So physical geography is important here.

Castle-building is no exception to locale meaning more than period; whether you're talking about Notre Dame or other medieval structures on Île de la Cité in Paris, or one of the 19th century fortifications like Fort de Villiers surrounding it, they're both build with the same beige limestone ('Paris stone') which is local to the area, and as a limestone easily worked. (working stone = shaping it)

Here in Norway, Sweden and Finland there's very little limestone in most places. Much more commonly you're dealing with hard granite, that's difficult to work. But there are lots of rocks of varying sizes lying about on the surface, deposited during the last Ice Age. So the simple thing to do was to gather these and build with them, with minimal working of the stone. Take Bromma Church for instance, an 12th century church which doubled as a defensive structure. The stones are about 30-50 cm on a side, note the large amounts of mortar between them; they've not been worked or fitted together neatly. If any part of the stone was worked it was to flatten the outside to make a smooth wall. (Note, these walls were historically limewashed and you would not see the stones. The ideal was for it to look smooth and homogenous. The idea of medieval castles with visible stones is from 19th century romanticism when people developed such a fetish for ruins that they removed limewash/stucco from castles that still had theirs. e.g. Örebro) Even runestones from the Viking Age often ended up being repurposed in walls. Not least since they typically had a side - the carved side - that was flat. Few large stones is more efficient than many small, but obviously there's a limit limit to how big stones that could be transported and maneuvered into position. Unsurprisingly, the largest stones are usually found near the bottom of walls. They're seldom larger than 2m on any one edge, and if they're that big they're surely narrow in the other dimensions. 30-90 cm is more like the typical edge size for these constructions. Here's my own photo of Kronoberg Castle. The handrail and such should give you an idea of the scale. Note how many of the stones have rounded edges, hinting that they were either taken out of the water or worn round by glacial streams, and not worked

But there are exceptions; Visby on Gotland and Tallinn in Estonia were important medieval cities which had local supplies of easilly worked limestone, leading to the impressive medieval fortifications still standing in these towns (Tallinn, Visby) Being easily worked it was easier to split these stones into smaller chunks or rectangular blocks which would fit more tightly into the wall. Some rock may be easily worked and good construction stone but the bedrock full of cracks, in which case you can't necessarily work much of it into rectangular blocks but just have to use the irregular pieces you get.

Not everywhere had a lot of rocks. Denmark and the coasts of northern Germany and Poland are largely flat areas of clay-rich soil that used to be sea bottom. So there wasn't a lot of random rocks poking up, and the bedrock was often far below (and not always a stone worth quarrying anyway). So they used the clay and made bricks and the 'brick gothic' style developed which you see in that area, such as Lübeck and Gdansk. castle building was no exception with e.g. the massive Malbork castle, built by the Teutonic order in northern Poland. Brick Gothic was also used in areas that did have stone, like Uppsala cathedral in Sweden. And the reason is that although stones were plentiful, there was a lack of easily worked stone. Although a chonky Romanesque church could be build with largely uncorked stones, the highly ornate and light architecture of the Gothic cathedral required something else. So the building was mainly built out of brick, with some marble (which existed in small amounts in the region) brought in for the finer details.

So to summarize the various options (not that this is a formal classification or anything): Bricks, mostly-unworked big rocks you'd find around, unworked quarried rocks, and quarried rocks that'd been worked into nice rectangular blocks.

If you jump forward in time and look at a modern castle like the iconic Neuschwanstein (construction began 1869) in Germany, it's largely built out of quite large (40x70cm) or so blocks of granite, which is a hard and not so easily worked rock, worked into very neat blocks. But this isn't very medieval-ish; it was made possible by the steam tools available in that era. (although it was still an extremely expensive building)

But the short of it, is that what material was used depends entirely on when and where.