Short answer: it's exactly how you describe it. The Crusaders were by and large way over their heads deep in a land populated by people that actively opposed them, and the Latin Empire's preeminence in the Byzantine corpse lasted about a year. At the same time, the Byzantine rump states were unable to provide a united front because in the aftermath of the halosis, or sack of Constantinople, the primary players disagreed as to who should hold the imperial dignity. The primary players were powerful Imperial family associates who were able to amass individual power bases under the structure of the Komnenoi and Angeloi-led Empire.

For the long answer, we need to look at the Empire's structure. In 1204, the Byzantine Empire was a remarkably centralized state, more so than any comparable state in Europe. At the same time, it had gone through profound changes that emphasized the importance of the military aristocracy and the Imperial family. You may be aware of the Thematic system that provided for the Empire's armed forces during the early middle ages. This system largely collapsed during the 11th century crisis, and Emperor Alexios Komnenoi was forced to come up with an ad-hoc system to create a force to defend his territory in the wake of the loss of central Anatolia. Faced with invasion on all sides, Alexios relied increasingly upon his family members and other powerful figures in the military aristocracy he came from. Most Byzantine aristocrats had a group of soldiers and other armed hanger-ons attached to their persons, or retainers. Without a central army, this was the best Alexios could muster. Further, facing a lack of manpower but with a significant amount of cash and specie, mercenaries became an extremely important part of the army. Though a small core army would eventually supplement these, retainers and mercenaries would be a key facets of the Byzantine army throughout the 12th century. For maintaining state cohesion in the event of a disaster, this is a bit of problem, as the mercenaries would leave at the first sign of trouble or lack of cash, and the retainers, while mostly indigenous fighters, would place their loyalty not in the Empire, but in the noble they followed.

The nobles that fought for the Emperor tended not to do so out of the kindness of their hearts, but rather the opportunity for personal gain. The Imperial family in particular vied for position, and a lack of favor from the Emperor could well lead to that family member claiming the imperial title for themselves, as in the case of Isaac II and Alexios III Angelos, or Isaac and Andronikos I Komnenos. To keep them loyal and happy, as well as provide the material to maintain their important retinues, the Emperors would increasingly grant pronoiai. For this, a pronoia is best understood in relation to its Western European counterpart, the feudal grant, though they differed significantly. The Emperor would make a grant of an Imperial resource, usually land, to an individual, who would be entitled to utilize the grant and all of its income. Unlike a feudal grant, these were extraordinarily rarely hereditary (though they lasted through the individual's lifetime, and accumulated wealth was certainly heritable), and came without an explicit obligation for military service- the Emperor would grant a pronoia to an already loyal noble with the expectation that it would help bolster that noble's military capacity. As in western feudal institutions, an individual could become quite powerful through these grants.

To be clear, by 1204 the Empire's armed forces depended primarily upon the retainers of powerful aristocrats and imperial family members as well as a copious amount of mercenaries. This was further complicated by the fact that the Imperial unity was at an all time low. Andronikos I Komnenos blew out the Komnenoi dynasty's legitimacy through his cruelty, and his successor Isaac II Angelos, through his heavy taxation and political bungling was for many hardly a fitting replacement. Eventually he faced a coup by his brother, Alexios III, which further destabilized the country and was an indirect cause of the Fourth Crusade, as Isaac's son, also named Alexios, would seek foreign support for his claim. By the time the Crusader's sacked the city and declared their intent to take over the Empire, very few Byzantine nobles agreed as to who should hold the Imperial dignity.

Without its capital or an Emperor, most of the Empire turned into a power vacuum that the Crusaders hoped to fill with themselves. The mercenaries left at first notice, and the retainers were following their nobles, who themselves had no idea what to do. The Crusaders were a relatively small force, but were able to use the disunity and nobles' paralysis to their advantage. They faced little organized opposition in Europe- the Byzantine army's presence in the Peloponnesian Peninsula, for example, was nothing more than a few garrisons and retinues. They were able to mount a resistance of about four thousand men at the Battle of the Olive Grove, but the better equipped yet outnumbered Crusader knights made short work of them, allowing for the establishment of the Principality of Achaea.

Asia Minor was a different story, primarily because the Crusaders had a minimal presence there. Make no mistake, the Crusaders wanted to subjugate the region, and had even intended on creating four duchies in what remained of Byzantine Anatolia. But the Anatolian aristocrats, by far the richest and most powerful of the Empire, weren't going to take that lying down. They were still organized, and ultimately able to unite against the Crusaders. Theodore Laskaris was an associate of the Komnenoi and Angeloi who left the city because of the crusade, and found himself in the best position to organize the Byzantine nobility in Western Anatolia. He did so, and many refugees, including the Ecumenical Patriarch, flocked to his capital in Nicaea. Theodore was proclaimed Emperor with all the force of Western Anatolia in opposition to the Latin Empire.

Given the distance and the fact that there were powerful nobles everywhere, Theodore was note the only one. Michael Doukas and his brother Theodore held various pronoiai and the governorship of Epirus, and were likely the strongest nobles in the west at that time. They initially declared their allegiance to Boniface of Montferrat, the crusader King of Thessalonica, but soon decided to revoke their allegiance to Thessalonica, becoming a major depository for various Byzantine refugees. This support proved useful, and the brothers caused the collapse of the Kingdom of Thessalonica through the following two decades. They hardly felt any reason to answer to the Empire in Nicaea, so Theodore, upon his conquest of Thessaloniki also claimed the Imperial dignity.

Trebizond is an interesting case, as they actually seceded a couple weeks prior to the sack of Constantinople. Alexios and David Komnenos, grandsons of Andronikos I, took advantage of the chaos of Constantinople's siege, seized Trebizond, and the former declared himself Emperor. This would have been a serious challenge that would have resulted in civil war for the Empire if there had not been a sack, but in light of the fact that it happened, Alexios and David's actions just fell into the general chaos of the halosis. David was able to secure the allegiance of the Byzantine nobility close to Trebizond, and that was that.

The Crusaders, despite some of their successes and the aforementioned Byzantine paralysis in Europe, still had trouble subduing the Byzantine nobility. It must have looked like a dream come true for the crusaders in establishing a new crusader state in Romania, but the Latins would have their wake up call when the Byzantine nobles, with no one else to turn to, would request the Bulgarian Tsar's assistance in defending their land. The Bulgarians made short work of the Crusaders, completely annihilating their army and capturing their Emperor less than a year after the Latin Empire's foundation at the Battle of Adrianople. The Latin Empire and the Crusader states in general would remain players in the politics of the region, but their preeminence, if they had any, was completely shattered. The combined pressure of Bulgaria, Epirus, and Nicaea would destroy them entirely in the north, with only Athens and Achaea managing to miraculously survive.

So in short, the Komnenoi and Angeloi Emperors made their family members and other nobles extraordinarily powerful with their own power bases, and depreciated the value of their office. When a crisis hit their capital, paralysis temporarily struck, allowing the Crusaders to temporarily make huge gains. This would reverse itself as resistance movements coalesced. Nevertheless, it was a terrible blow. Michael VIII Palaiologos would recapture Constantinople, but would find most of the old city abandoned. The Empire would never recover the strength it once had, and was already entering decline even at the end of Michael's reign.