An Average castle was quite small. Let's look at the castle of Lussan in Gard, in the South of France, near Uzes.

The town of Lussan has 480 inhabitants today. Probably about the same number back in Medieval times. It is a slightly larger village than the average village in the Uzege, but not as large as the larger towns.

Every one of these little villages (circa 30 villages within the same radius (20km) from Uzes as Lussan) has a castle, mostly built in the 12th century. (The castle of Lussan was first built in the 12th century, though it was modified over the years, and its current appearance is more or less as it was in the 15th century.)

Here are some pictures of Lussan, a satellite view, with the position of the castle marked. Two pictures of the castle, and a picture of the walled hilltop town of Lussan (the castle guards the entry of the road up into the town).

http://imgur.com/a/WVXXK#P81mJCe

If you look at these pictures, you can see that the castle footprint and height indicate that the castle is about the size of 8 of Lussan's normal houses.

If we guess that it had some features that the normal houses didn't have (its own well, thicker walls, etc.) Then, perhaps it cost as much as 10 or 12 normal houses.

The town of Lussan has some 80 or so houses, so the castle cost about 15% of the total cost of all the buildings in Lussan. Except, this did not include the town wall (which might well have cost more than the castle).

Another way to look at it is that with a population of 480 people, Lussan probably had a workforce that could be assembled for castle building of circa 100 people for short periods of the year (after harvest, before planting).

With a few masons, carpenters and building specialists working full time on the castle, year round, and a work force of about 100 working for 6 weeks in the year each (40 days was a traditional service period in Medieval times). The village might have spent about 870 man weeks a year building the castle.

If it took two years to build the castle (that includes quarrying and shaping all the stone, felling and trimming all the lumber), then that might be circa 1740 man weeks of labor. Or about the same as 20 man years per year for two years.

The labor would be the main cost to the village of building the castle. They had the stone quarries on their land. They had the trees in the local forest. For a couple of years they had to devote their collective service and some of their stone and lumber resources to building a castle rather than building houses, or a mill, or barns, or maintaining the roads.

The small castle of Lussan would have been the home of one knight (the local seigneur) and his family and servants.

Of course, there were many larger and grander castles, royal castles and great fortresses, which cost much much more to build, but the average small castle might have cost the effort of all the service of a village for about two years.

What do you mean by an average castle? What century?

Let's look at the conflict between Edward I and Wales, which led to the construction of some of the most impressive castles in the UK, the "Iron Ring" around Wales. Edward's wars in Wales were ruinously expensive. England's annual tax revenue was around £26,000 per year (and lots of fun information about his taxation can be found on that link). But his massive army was so large that it fragmented Welsh resistance, so it served its purpose.

To avoid having to deal with the Welsh again, Edward spent roughly 80,000 pounds building ten castles.

The most expensive were 15,000 pounds for Conwy, and 25,000 pounds for Caernarfon.

Garrisoning them would be relatively cheap, since they only needed 30-40 soldiers to hold them, so perhaps 6,000 pounds per year to garrison in peacetime. Certainly much less expensive than fighting another war.

Unfortunately, it almost didn't work out that way for Edward, as the castles were besieged by the resurgent Welsh. Caernarfon was captured by the Welsh, but the remaining castles held out long enough for the English to relieve them.

The Irish Tower House, amongst the smallest of structure we'd call a castle, may have been built for as little as £10. In 1429 King Henry VI introduced a measure granting £10 to any man in the English controlled section of Ireland (the Pale) who would construct one. If one assumes the flurry of tower house building (several thousand) was set off by this, the subsidy probably covered a large proportion of the cost, though by no means were they only built by English loyalists. For those unfamiliar, these were relatively simple rectangular plans of 4-5 stories and a minimum of 20'x16', often with a small curtain wall.

Source: Sweetman, P.D., 2005, Medieval Castles of Ireland

Its is a bit off topic but when generally pricing medieval stuff you can actually look towards today's prices from a black smith who produces medieval goods. Not a mass market guy but something like a made to order sword can easily go up to a thousand dollars, singular pieces of armor as well. The price was only slightly less back then but nowadays because there is no real big market for the stuff and there is hardly anyone making it, you can rough gauge based on man hours, and materials.