The two main reasons have been already presented by u/BRIStoneman, so let me delve a bit deeper into the realities of the medieval life to point out why it would have been impractical to build very large structures in Europe.

Castles were build by local landowners who treated them as the residential buildings, administrative centres, local treasuries and garrisons and the places that could have be used to deter potential troublemakers, such as bandits, rebels or covetous neighbours. And thus they were relatively small, because they were made according to the necessities of the landowners's families and their immediate subjects living in the nearby villages. Furthermore, a network of castles and later also cities served a defensive purpose by itself, forcing the potential enemies to move in more or less predictable manner and dividing their forces to engage a substantial part of local forces. But as already mentioned, the main purpose of the castles was usually not a strictly military one and the structures themselves were not a main target during an invasion..

Let's start from the issue of self-sustaining castles. We can safely assume that the average medieval village consisted of roughly 40 hearths (households) (in 14th century Germany a village with 20 households was considered 'small' and one with 60 households was, conversely, big) thus housing close to 250-300 people, including children (I'm using Herlihy's calculations of 6 people per average family in medieval England). Now, an average English household usually held the land equal to one yardlength, equal to roughly 30 acres defined as 40 by 4 perches (5,5 yard or 5 meters). This means that every household had roughly 145.200 square yards or 12 hectares of land. This corresponds to the situation in Germany or Poland in the corresponding times, where single household rented a land area equal to 10-16 hectares while 25 hectares was considered a substantial amount, capable of easily feeding the family and leaving something for trade. Now, assuming that the all households are roughly equal economically (which is a simplification as mayors could have been holding 100 acres of more), this gives us the area of at least 480 hectares for a single village, with the part of land used for meadows under three-field system already factored in.

For the ease of representation, let's assume that the land is contained inside the perfect circle with an area of the aforementioned 480 hectares or 4.800.000 square meters. Using a basic planimetrics we know that this gives us a circle with a radius of 1236 meters and the circumference of 7764 meters or 4.8 miles. Assuming a small but sturdy wall surrounding that territory, roughly 5 meters tall and 2 meters wide (16 x 7 feet, give or take), then a 1 meter section of such wall would equal to 10 cubic meters of stone. And to encircle aforementioned land with such a wall, one need 77640 cubic meters of stone. As the average density of granite is 2.7 g/cm^(3), the total mass of stone to be cut, transported and assembled amounts to 209.628 tons. And this is just for a small wall protecting land nominally allowing the sustenance of 250 adults. And with 4.8 miles of length, this would have still been far longer wall than any medieval defensive wall built in Europe. For comparison, Dubrovnik has walls slightly short of 2 km (1.4 miles), Avila has 2.5 km (1.6 miles) and famous Carcassonne has walls 3 km (1.85 miles) long. This, of course, does not mean that longer walls were not built, far from it. Aurelian Walls encircling Rome were over 19 km (12 miles) in length and fortifications protecting Chinese cities like Beijing, Zhongdu and especially Chang'an were easily over 24 km (15 miles) long. But we're speaking of the largest cities around, serving as capitals to the powerful states, not a small defensive points. Just to drive the point home - to encircle the area capable of feeding 1000 people using the Medieval agriculture techniques, one would need to build walls larger that the largest defensive walls ever built around any settlement in history. This quite clearly shows why no one ever thought of fortifying fields.

Such walls would have been also extremely hard to defend. Assuming that they were indeed self-sustaining fortifications, in the example above we have the 7.7 km wall encircling the fields able to sustain 250 people. Even if we assume that all these people are able to competently defend, if all they stood on the wall keeping equal distances between them, they would have been standing 31 meters apart. This is a very sparse defence, making any coherent action against the attacker nigh-impossible.

On the other hand, did no one build a fortification housing 30.000 soldiers? Of course they did, just nowhere close to the Middle Ages. The largest European fortresses built in late 19th century, such as Verdun in France or Przemyśl in Austria-Hungary (now Poland) were even larger, being able to house more than 120.000 soldiers and over one thousand pieces of artillery with corresponding ammunition stores. But such fortifications were a response to a large armies that had a manpower in high hundreds of thousands if not millions of people that relied on infantry and firepower rather than cavalry and were organized centrally, what has been possible only after the dawn of absolutist monarchies. Such a manpower was maybe not unthinkable but generally unfeasible for any military power in the Middle Ages. With the relatively high fragmentation of power and the prevalence of feudal mechanisms regulating the land ownership, medieval armies were composed chiefly of well armed and armoured men-at-arms, usually knights and their retainers, for whom military service was a part of their economic obligations. A whole provincial military forces that could have been mustered by a high-ranking noble (owner of an average structure we usually associate with the expression 'medieval castle') called to arms by a monarch could have counted maybe 500 such warriors. Please note, that even in the late Middle Ages, armies fielded by European kings could have reached low tens of thousands, like in the case of Tannenberg in 1410 (21.000 Teutonic Brothers and allies against 30.000 Poles and Lithuanians), Agincourt in 1415 (6000 English vs 22.000 French) or Varna in 1444 (22.000 Christians versus 45.000 Turks). These were armies formed from the combatants from the equivalent of an entire country. No local polity could have boasted such an army on their own. At least not in medieval Europe.

So, to sum it up, the medieval castles were generally small, because their military role was auxiliary at best, the medieval polities were generally small and their economies were based on the personal agreements, resulting in relatively small if professional military forces that were based on cavalry (meaning that the manpower was best used outside the walls rather than during sieges) and the role of rich, well-defended places worthy of besieging being largely filled with cities rather than castles. But with the changes in the organization of the society that also resulted in the different way of army organizations, castles and later also cities gave way to modern fortresses and fort networks that enjoyed some popularity and then faded away in the age of modern mobility doctrine, aviation and long-range combat.

Ok, so there are two main issues here. The first is what the purpose of a castle is, and the second is one of sheer practicality.

Castles typically aren't meant to shelter a population. That's what town walls are for. Castles are elite residences designed to be a fortified base of operations from which the lord can project political and, if necessary, military power over a much wider area through the use of highly mobile cavalry garrisons. Castles, especially outside of the Crusader States, are rarely meant to stop an invasion as much as they are to deter and delay it, so that other forces can be marshalled to meet the enemy in open combat. Medieval battles are often quite small affairs by modern standards. The English army at Alnwick in 1174 was only some 3-400 men, gathered from the garrisons of nearby castles. In fact, this battle illustrates how castles worked. The Scottish invasion had been repeatedly delayed by small English castles (such as Brough with its garrison of about a dozen knights) and had finally laid siege to Alnwick, but in spreading its forces to do so, left itself vulnerable to a localised attack by the garrisons of Newcastle, Prudhoe and Carlisle, who were able to overwhelm the 60 or so men with the Scottish king William and capture him. Robert de Beaumont's army at the Battle of Fornham was around some 3000 men, while the loyalist army under Richard de Lucy numbered some 300 knights (plus assorted implied men at arms). The strength of knights was not on a rampart but in the saddle.

Consider some practical reasons: a castle the size of even a modern city, let alone a county or whole country, would be unfeasibly large, slow and expensive to build, before you even consider the staggering manpower costs necessary to garrison it, especially in times of peace. Consider as well the need to build multiple internal defences in order to not rend the entire endeavour immediately impractical should a single breach be made.