The empire itself, in many cases. The only reason Manzikert happened was because of the incompetence of the emperors between Basil the second and Alexios Komnenos. Under Constantine the 10th in particular, the Thematic System was dismantled and replaced with mercenaries. This was done because theme armies had in the past elevated their Stratigos to the purple, thus Basil II's successors saw them as a threat to their rule. In turn, this caused a steep decline in the ability of the army to defend the Cappadocian and Armenian frontier, and the political instability and civil war following the exile of Romanos IV allowed the Turks to settle almost unopposed.

The Crusader sacking of Constantinople was successful for many of the same reasons. After the death of Manuel Komnenos in 1180, emperor after emperor was usurped by another rival, and once again the institutions that the Komnenian Dynasty had painstakingly repaired saw a sharp decline as political instability, the gradual loss of imperial territory to invader and rebellion, economic domination by Italian Merchants and the increasing power and independence of the Aristocracy undermined the Imperial system. The army of Alexios, John and Manuel had declined to such a state that the only professional unit present in Constantinople during the sack was the Varangian Guard, with much of the army having been decimated for lack of funding and neglect.

In truth, The biggest cause for the collapse of the Eastern Romans were in fact the Romans themselves. The empire had survived and bounced back from far worse crisises then Manzikert and the Latin Empire, but it needed capable leadership and political stability for that, something which by the time of the Fourth Crusade the Empire had lost for good.

Sources:

Byzantium-The decline and Fall By John Julius Norwich

The Development of the Komnenian Army 1081-1180

Byzantine Wars by John Haldon