(Part 1/2) This is a topic that I have covered in some of my own introductory lectures to pre-modern European history (before 1800), and I'd be happy to repeat some of my points here. The important thing to note is that it’s not easy to provide a single explanation for the development of pre-state polities into states. The reason for this is that wasn’t one event, but a process of individual but cumulative decisions which collectively transformed kingdoms into states. To look at some of the reasons, I'll do a case examination of England, though with occasional digressions to other kingdoms too.
What is a state? Historians struggle a bit with the concept of state. Many have argued that the word has too many modern connotations to be used on most pre-modern societies, while others use it to describe almost any historical and modern governmental structure (polities). Others, including myself, would argue that there were many pre-modern states, although most pre-modern polities were not states. Just like modern states, these pre-modern states had a stable bureaucracies, comprehensive systems of taxation, and professional standing armies. With this definition the Roman Empire, the Byzantine Empire, and medieval China would all be states, and probably a few more polities too.
The go-to definition of state, though, is that of Max Weber. He defines the state as "a human community that (successfully) claims the monopoly of the legitimate use of physical force within a given territory." (From the lecture “Politics as a Vocation”, 1918). I don't know enough about the Roman, Byzantine, or Chinese empires to say whether they would fit this definition. But as your question assumes, no European polities (apart perhaps from the Byzantine Empire) were in fact states in the Early Middle Ages.
The pre state polity Until the High Middle Ages (ca. 1000 - ca. 1300) Europe was dominated by what we can call pre-state polities. In some places the phase lasted much longer. These polities had some common features: Power was based on personal connections such as friendship, loyalty, family, etc., and for this reason kings and other magnates often had to be personally present in the localities to exercise rule, decide on lawsuits, deal out punishment, and to establish and confirm alliances with local magnates through marriage, wardships, gift exchange etc. Pre-state polities also have few or no governmental institutions, use little or no writing, and legitimise power by inheritance, conquest, or religion. The system has been beautifully explored by Hans Jacob Orning in Unpredictability and Presence: Norwegian Kingship in the High Middle Ages (2008). A more classical work is perhaps Gerd Althoff's Family, friends and followers: Political and Social Bonds in Early Medieval Europe (1997).
That power relied on personal relationships, the presence of the ruler, as well as on the competence and personality of leaders meant that successful polities rarely remained successful for very long. The empire of Charlemagne (r. 768-814) fell apart after his death, and the same happened to the North Sea Empire of Cnut the Great (r. 1016-35), and the Angevin Empire of Henry II (r. 1154-89), which fell apart under his son King John (r. 1199-1216). In states (modern and pre-modern), strong institutions make sure the business of government continues regardless of the competence and personality of its leaders. This explains why the Byzantine empire was able to remain relatively powerful for centuries, even though many of its leaders were outrageously incompetent or violently overthrown.
Growth of Administrative institutions During the High Middle Ages many European polities slowly developed their administrative institutions towards becoming modern states. One of the most important developments happened in the use of writing. Norman England was early in many of these developments. It is a good example of a pre-state polity taking its first steps towards statehood.
William the Conqueror took over a relatively well-developed administrative from the Anglo-Saxons who probably had used writing more than anyone in Europe apart from the Byzantines. But the conqueror still had to travel personally around England and his French possessions to quash rebellion and confirm his rule. But in 1086, William the Conqueror utilised the Anglo-Saxon administrative system of shires, hundreds and shire reeves (sherrifs) to make an impressive survey of properties. The resulting Domesday book was actively used as a work of reference in court and other contexts to determine historical rights to land. It gave the king and the people predictability. The Exchequer was probably established in the early decades of the 1100s, though possibly earlier, and exacted taxes and land fees while documenting the transactions in pipe rolls extant from 1130. This ensured and documented the king's income.
Later in the century, Henry II standardised legal documents to make the judicial system more efficient and available. He also instituted the circuit system of itinerant judges who met in London to discuss common standards for a variety of cases. This eventually developed into the English Common law. All of it made the business of government, finances, law and justice more available and predictable, and less dependent on the whims of the monarch.
But another important reason for the development of these institutions was that English government had to work when the king was away. Apart from King Stephen (r. 1135-54) and a few years under Henry I all English kings from William in 1066 until the end of the Hundred Years’ War in 1453 had substantial possessions in France. Since the king often had to assert his power by being present in the localities he was often away from the kingdom, sometimes for several years. William I and Henry I had relied on their wives to run government when they were in France, but later kings would increasingly rely on a growing bureaucracy and appointed officers such as the Chief Justiciar.
With the issuing and renewal of Magna Carta in the thirteenth century, and the development of Parliament, an arena for cooperation between king and aristocracy (and later lager swaths of the population) was established. Civil wars were fought over these institutions, but eventually the idea of Parliament appealed to kings, because they could raise money relatively easily. In fact, Parliament turned out to be so effective that, during the early years of the Hundred years’ war, the English king raised almost twice the amount of silver (92 tonnes in 1339) than that of the king of France (53 tonnes in the same year), even though France’s population was 3-4 times that of England (13-17 million vs 4-5 million ca. 1300).
The Hundred years’ war War forces modernisation because effectiveness are always prioritised over other concerns. War clearly determines power over territories, and defines nationalities and identities through deciding on friends and enemies. American sociologist and political scientist Charles Tilly has studied modern state development and famously said that “war made the state and the state made war.” This seems very much to be the case with France and England during the Hundred years’ war.
As I said, the English king’s cooperation with Parliament enabled him to outspend the French king in the first stages of the war. This is surprising, because English kings were in theory less powerful than French kings, not only in terms of the population, but French king also had the power to exact taxes at will, while the English king had to ask Parliament. But in practice cooperation worked better, because the country was more willing to give up money through taxes when they get something back. For unlike the French, the English could ask the king for concessions before they allowed him the power raise money for war.
But the king/parliament cooperation is still a personal relationship that depends much on the personality and competence of the king. In the early 1400s, it seemed like the war was won with the successes of the energetic Henry V of England. But when he suddenly died in 1422 he left the claim to the English and French thrones to his 9 month old son. With this France went on the offensive. Jeanne d’Arc’s miraculous victories 1429-30 fuelled French patriotism; Charles VII of France made the taxation system more efficient, abandoned the feudal system and created a professional standing army. After having entered a new alliance with Burgundy, France turned their fortunes around, kicked the English out of the French south and took huge leaps towards a modern state.
(Continues below)