Castles were more frequently built of stone than of brick. (Though some brick castles were built, Malbork Castle, built by the Teutonic Knights in Prussia, in 1406, is the largest brick castle ever built.)

http://en.wikipedia.org/wiki/Malbork_Castle

Before castles were built of stone, they were often built of wood in Northern Europe (motte and bailey castles).

http://en.wikipedia.org/wiki/Motte-and-bailey_castle

Castles built in stone were built using many different techniques in different places and at different times. Generally the walls were built with an inner face and an outer face of laid mortared stone, with the space inbetween filled with a mix of mortar and stone rubble.

The ground floor of the castle might well have a barrel vaulted ceiling (requiring the use of wooden forms to support the vaulting when it was being laid, which would later be removed). (Other forms of vaulting besides barrel vaulting might also be used.) Higher floors could also be vaulted, but more usually would have wood, stone, terrazzo or tile floors laid on heavy wooden beams.

Wooden scaffolding would be used by the stone layers once the walls were high enough above the most recently built floor to make working on them from ground or floor level impractical.

Roofs, in Southern Europe, were most frequently Roman Tiles, while slate or other materials might be more common further north.

Dressed and shaped stone would often be used around gates, doorways, arrow slits, and windows.

Castles were generally built from material which could be found or produced very near by, as transporting large amounts of heavy stone over long distances was impractical in Medieval times unless it could be done by barge or ship on navigable water.

Brick was probably used when a castle was desired in a place that was far from suitable stone quarries, but near to deposits of clay and a source of fuel for brick making.