Why exactly did Italy break into smaller city-states?

Tiako

There isn't really a single "why", because what you are asking about is essentially a process that lasted at least five hundred years, and would argue longer. It is first important to remember that this process was by no means predetermined: the Ostrogothic Kingdom in certain respects maintained the Roman state apparatus better than the other successor kingdoms. The best evidence for this is, in many ways, the sheer vigor with which they were able to resist Justinian's conquest. Despite the resource base of the Eastern Roman Empire and the brilliant leadership of Belisarius, it took twenty years of brutal on and off warfare to subdue the Peninsula, which depleted their resources to such an extent that the Lomards were able to conquer much of Italy soon after. The Lombard Kingdom was also fairly well organized, although the brutality of decades of warfare left them unable to recreate a state with as much strength as the Ostrogothic one, and they were unable to fend off the Frankish Empire under Charlemagne.

So the problem is not that Italy has some sort of inherent character towards disunity that experienced a brief period of unity under the Romans. The problem, as you may have inferred from my previous paragraph, is that it kept getting invaded. If you look at a picture of Italy during the ninth century, you would see areas (particularly in the south) under the the Byzantines who were increasingly unable to assert control, northern Italy under control of the Franks and thus drawn into the particular drama of that empire, and de facto independent Lombard states in the middle. Italy was something of a playground of empires during the early Middle Ages, but as Machiavelli observed regarding the Papal states some centuries later, they were only strong enough to prevent others from unifying the peninsula.

A lot of this is taken from Christopher Beckwith's The Legacy of Rome, which pays careful attention to isses of state power and centralization.

AlviseFalier

Like other answers said, it has to do with Roman heritage; the reasons are varied. Will that stop me from giving an overview? No!

Part I: The first two kingdoms of Italy (with Justinian's interruption)

As your question leads me to believe, you already know that Italy was only a unified kingdom for little over two hundred years after the fall of the Roman Empire. However, the Kingdom of Italy, re-established after Justinian's reconquest by Lombards (who by this time had pretty much gone native) had always clashed with Byzantine Empire, which never really gave up on reconquering Italy, although the Byzantines could only really hold some coastal cities, while the fertile heartland of the Padanian Plain was firmly in the hands of the Lombards, till the Lombard King Desiderious (d. 786) got very close to finally kicking the Byzantines out of Italy; a successful offensive saw the encirclement of the Pentapolis (the five major Byzantine strongholds along the Adriatic Coast that to this day are called "Romagna" since the Byzantines were the last "Romans" to leave) which allowed Desiderious to march south against Rome unopposed. With the Byzantine army cornered in the Pentapolis and fearing a loss of independence for the bishop of Rome, who had been pretty much guaranteed a free hand to rule the "Duchy of Rome" (modern Lazio) in exchange for nominal Byzantine rule and military access, Pope Adrian cried "Uncle", excommunicated Desiderius, and asked Charlemagne to come to his aid. Charlemagne, even though he was busy conquering what would later become Germany, was immediately flattered, and marched for Italy.

Part II: Charlemagne

Charlemagne moved south in two columns, one under his personal command taking a westerly route, through the Valley of Susa, and another under his uncle, Bernard, through the Valley of Aosta. An ancient system of fortifications called the "Lombard Locks" (Chiuse Lombarde) sealed off the valley of Susa, defense personally commanded by King Desiderius held fast against Charlemagne's assault. However, the Valley of Aosta was guarded by Desiderius' son, Adelchi, who had been entrusted with a much smaller force, and was beaten back by the enemy at Ivrea, forcing him into a disorderly retreat. Desiderius had no choice but to withdraw to the fortress of Pavia to avoid encirclement, while his son fled all the way to Verona. Soon, Desiderius capitulated and would spend the rest of his days in a Belgian monastery; Charlemagne was crowned king of Italy, and Adelchi fled to Costantinople (which, in adherence to the principle, "The enemy of my enemy is my friend" saw the Byzantine emperor welcome the deposed prince with open arms now that Charlemagne was claiming the title "King of the Romans"). After Charlemagne's conquest of the Lombard Kingdom in 774, Northern Italy was divided among several Ducal Families, mirroring the feudal structure in the rest of Europe. Among these, some of the most important were the House of Obertenghi (later called the House of Este) dukes of Lombardy, the House of Savoy, ruling most of northwestern Italy, the House of Sambonifacio, who were Marquis of Verona, and the House of Canossa, Marquis of Tuscany.

Part III: The Feudal Lords get absorbed into the urban system

However, these feudal lords took less than three hundred years to become irrelevant; eclipsed by the cities. Unlike France, Spain, Germany, and England, Italy was extremely densely populated with Roman cities. How did Italy maintain urbanization? Being the cradle of Roman civilization certainly helped; the Barbaric Invaders were not so much Barbaric "Invaders" but barbaric "Imitators", and almost immediately adopted the "Roman" way of life: the founder of the first Kingdom of Italy, Theodoric the Great, was an ethnic Ostrogoth whose kingdom was propped up by the Byzantine emperor Zeno. Theodoric himself had spent his whole youth in Costantinople, by some accounts he spoke better Latin than Gothic (and his first language was Greek). The Lombards, who replaced the Ostrogoths, instituted a stricter, classically "Feudal" social structure, but their conquest of Italy occurred only after the Byzantine emperor Justinian betrayed the Ostrogoths and reconquered Italy, further sustaining the "Roman" way of life, making true feudalism impossible. After Charlemagne conquered Italy, the cities continued to be centers of life, rendered all the more important by being seats of Bishops, who often acted as points of authority for local lords. The wealth of these cities meant public life was never really centered around the household of a single feudal landholder; in Milan, for example, the landed families of Della Torre, Argentea, Baggio, Busto, Grassi and Caronia, along with many other smaller ones, all converged onto the Bishop of Milan, asking for privileges in the marketplace, arbitration of disputes and in a general sense all the functions that would normally be rendered by the Obertenghi, who were their feudal overlords on paper, but were becoming rapidly irrelevant beyond their castle in Este. Another factor that accelerated the irrelevance of the Italian Dukes and Maqruis was the fact that the "point of reference" or sovereign, was far away across the alps, making their feudal obligations increasingly irrelevant. Indeed, it took less than a hundred years for the title of King of Italy to be disassociated with the crown of Germany; and create a sort of feudal anarchy, where the title King of Italy was continuously contested between the various large landholding Italian families. Otto I tried descending into Italy and clean things up again, but this only resulted in the definitive irrelevance of the large Italian Lords. The small Italian counts and barons, for their part, had stopped caring.

Part IV: The Communes

In many Italian cities, the convergence of so many (locally) wealthy and powerful individuals in the cities, along with the influence of an "Artisan" class that could only exist thanks to the density of urbanization, meant that many cities adopted a unique, vaguely democratic system of government. Although the system changed from city to city, most established a legislative council of leading families, and often, the leaders of local artisans of merchants guilds were allowed to participate as well. Many cities elected a Podestà, who was a leading citizen of another city to act as an arbiter. Thus, by the Year 1002, the cities of Crema, Cremona, Milan, Monza, Bergamo and Brescia were all thriving "Communal Republics", and the House of Obertenghi, who should of ruled over them, was all but unknown! Along with the Podestà, the most important job in the communes was that of Capitano del Popolo, or Captain of the People. He acted as a Commander-in-Chief when the city was threatened with war by a rival city, and commanded the cities' forces should they decide to conquer a neighboring city. Milan, Verona, Florence, and to a lesser extent, Padua, gained a reputation as "Bully" cities in northern Italy, expanding their territory far beyond their outskirts.

Part V: The end of the Communal System

While the early form of communal democracy was good when all families were on equal footing, what happened when one was much more powerful than the rest? Starting from the mid 13th all the way up to the early 15th century, there was a trend for one family to dominate the others, and either pay lip service to the communal institutions while really ruling alone (like the Carrara in Padua, the Dovara in Cremona, and famously, the Medici in Florence) or sometimes outright dismantle them, often via bloody internal struggle against opposing families (like the Visconti in Milan, the Scaligeri in Verona, and the Ezzellini in Treviso and Vicenza). Keep in mind, it's not like these families had dropped everything to move to the cities; they were still feudal landholders, who held large estates and castles outside the cities from which they could call their own vassals to arms; many powerful lords, such as the Visconti's rivals, the Della Torre, were feudal lords of entire fortified burghs in addition to their various estates. Although they might draw as much income from trade as from rents and agriculture, the two things are linked; you need capital to start trade ventures, and society was still mostly agricultural. By the early 15th century, even among those states that had maintained their communal government, there was no doubt that the Italian cities had all the trappings of established states; with definite borders, internal institutions, and embassies. And they were not necessarily city-states at that; many ruled over large stretches of territory, which although were small compared to the other European states, often surpassed them in wealth; troubled European rulers (especially French Kings) had a habit of marrying Italian princesses when they wanted to strengthen their treasury.