A similar question to this has come up before in relation to why stone building declines with the Roman Empire, and the answer is broadly the same: logistics.

The Roman Empire didn't end overnight but broke down slowly in a series of civil wars, revolts and crippling recessions which slowly gutted the ability of the Empire to maintain effective provincial control. An often-overlooked facet of the Pax Romana which creates perhaps the most havoc with its decline is the huge commercial network of trade routes and productive sites which span the Empire. The Roman Empire is built as much on trade as on conquest, and settlements are planned with trade in mind. The Pax Romana allows for the establishment of riverine and coastal settlements, vast agricultural villa complexes manned by armies of slaves, and perhaps most importantly, provincial specialisation based on economies of scale. This meant, for example, that Italy and Gaul could focus nearly their entire agrarian efforts on the production of wine, or Greece on olive oil, safe in the knowledge that the vast grain requirements of the population would be supplied by the farms of Iberia and North Africa.

As recessions bite, the economy slows and instability breaks out in the provinces, this entire structure begins to fall apart. Revolts in North Africa prove particularly problematic for the Empire, and the regular delivery of vital commodities becomes less assured at the same time as those commodities become increasingly out of the price range of those who can afford them.

At the same time, diminishing security and the rise in raiding, invasion and general warfare as Imperial forces are stretched increasingly thin and even fight each other, so populations increasingly abandon low-lying settlements to return to the pre-Roman hillfort settlements that at least granted some degree of protection. Without a functioning logistical network and with Rome increasingly unable to provide governance, populations fall back increasingly on subsistance agriculture and pre-Roman organisational structures. This leads to a massive decline in populations and societies intrinsically become more rural and isolated.

In areas like England, this contributes to the initial decline in education. As the Anglo-Saxons take increasing advantage of the Romano-British decline to establish their own kingdoms, so Christianity is rebuffed and replaced by an illiterate Paganism. In much of the former Empire, the Church remains the last bastion of Imperial organisation and hierarchy, and as such is quickly sublimated into forming the basis of government structures for newly emergent successor states. This need places strain upon the church to divert much of its clerical activity into secular matters. A decline in classical literacy follows essentially because Ancient Greek is really, really difficult and few have the luxury of time to do so.

Some reading:

Mireille Corbier, "Coinage, Society, and Economy," in Cambridge Ancient History: The Crisis of Empire, A.D. 193–337; Harris, "Trade," in the same.

Charles-Edwards, After Rome

Bartlett, The Making of Europe

Garner, Hillforts of the Cheshire Ridge