The short answer is that after the death of Manuel I Komnenos in 1180, the Empire endured a long string of incompetent (or in the case of Andronikos I, deranged) rulers that severely undermined Byzantium's position as the first-rate superpower of the Mediterranean and the Near East. In the last years of Emperor Manuel's reign, there had been a series of setbacks in commercial and diplomatic ties with the Latins (specifically the Italian powers, most prominently Venice and Genoa) that made Byzantium look bad in the eyes of the West. Manuel, though I would argue an effective Emperor and loved by the Crusaders, unfortunately failed to secure the Imperial gains that he sought during his reign, and so Byzantium's reputation as a supremely powerful military force was somewhat lessened by the time of his death. Had there been an Emperor even half as effective as the previous three "Good Komnenoi" that succeeded to the throne in 1180 (and it's hard to say how well Alexios II Komnenos would have done had he been a little older when his father died), the Empire probably would have been fine. The military under Manuel was strong, and even though the Battle at Myriokephalon in 1176 was seen as a disaster, it does not seem that the Empire's back was utterly broken like it had been at Manzikert in 1071. A competent and lively Komnenian ruler following Manuel would certainly have been able to rally the Empire's forces into another offensive, which the Turks would have in all likelihood lost, restoring the Empire's reputation. The situation with the Latins could have easily been mended with prudent diplomacy and a show of military force, as it always had been. Unfortunately, it was not to be.

Andronikos I, with his obvious anti-Latin policies, allowed the so-called Massacre of the Latins to occur, and purposely destroyed ties with the West at this crucial moment in the Empire's history; he signed Byzantium's death warrant. His paranoid attempts to secure power by persecuting and brutally murdering his political opponents made him hated both inside and outside the Empire. While his goal in decreasing the power of "the powerful" was probably the right course of action, it was done in so completely the wrong way that it ultimately mortally undermined the authority of the Imperial throne. Dissent, rioting, and rebellion became more and more common in these later years of the Empire. The Normans, seeing the weakness and instability of the Empire, invaded a second time (the first was in 1081), but there was no Alexios I Komnenos to stop them this time. They ravaged their way across Greece, essentially unopposed, and sacked Thessalonika. And still, Andronikos seems to have continued his grisly persecutions, ignoring the problems that befell the Empire. His death in 1185 at the hands of his own people came not a moment too soon. And yet, the damage had already been done -- ties with the West, steadily improved during the years of Alexios I, John II, and Manuel I, were destroyed in essentially 3 years. The Angeloi (the successors of the Komnenians), a weak and incompetent family of rulers shown to be unfit to rule the Empire not long after their accession, drove the Empire further into a death spiral in the next 20. The army was neglected, diplomacy was neglected or conducted appallingly poorly (especially during the events of the Third Crusade), and the Emperors continued to ignore the problems so continuously mounting against them, choosing instead to retreat to the pleasureful confines of the Imperial palace.

When the Crusaders arrived in 1203, they found that the once-powerful Empire had essentially no navy, a less-than skeleton defense force defending the Theodosian Walls (which were, surprise, in disrepair), and having lost many Imperial lands to revolts (notably Cyprus and Bulgaria), it was obvious to the Latins that Constantinople was ripe for the taking, and a much more lucrative and easy to acquire prize than having to march into the death-trap of Turkish Anatolia or Ayyubid Syria, Egypt, or Palestine. Furthermore, the large Italian contingent of the Crusade (spurred on by the Doge Enrico Dandolo) sought revenge for the Massacre of the Latins under Andronikos I. It was only a matter of time before such incompetence (whether born through villainy or ignorance) was to be repayed in kind.