Yes, Byzantinum controlled less territory in 1204 than at its height; but it was also attacked because it had become TOO successful. Expansion into Italy, Hungary, and along the Dalmatian coast upset many of its former allies (e.g., Venice) and frightened others (e.g., Germany). Add to that a resurgent economy, which gave it a false confidence, threatened Venetian economy hegemony in the Adriatic, and made it a more attractive target. The Massacre of the Latins in 1182 was the result: Decades of brinkmanship and bitter competition exploded in an orgy of violence, blooding (as Runciman puts it) a Western European coalition against the city.

Angold is good, but see also Magdalino, The Empire of Manuel I Komnenos (for more on Byzantium's territorial ambitions in the West) and Harvey, Economic Expansion in the Byzantine Empire (for more on urban revival).

One major issue not already covered was the turmoil within the Empire leading up to the events of the Fourth Crusade, both in a general sense and with regards to repeated coups/revolts and an inability of the various emperors during the time period to provide adequate leadership.

Isaac II Angelos had taken the throne in 1185 with the support of a popular revolt, but despite some initial military successes against the Normans proved increasingly unpopular due to his taxation and increasingly unsuccessful military endeavors (he also did things like antagonize Frederick Barbarossa). Following more setbacks while fighting the Bulgarians, Isaac was overthrown in 1195 by his brother Alexios III and blinded and imprisoned.

Alexios III was unable to make much of a positive impact as a ruler, either - similarly unsuccessful in fighting the Empire's enemies and also mismanaging the imperial finances. Isaac's son, Alexios IV, also managed to escape his imprisonment and (per popular tradition - I believe the exact deal and context is somewhat debated) helped motivate the Fourth Crusade to turn on Constantinople by promising compensation and support for the Crusade if it helped him reclaim his father's throne. The Crusaders had not captured enough wealth after sacking Zara to appease the Venetians, so they were able to partially cloak their motivations to their army by claiming to be restoring the rightful ruler of the Empire to his throne.

Alexios III's conduct during the Crusade didn't help matters - he was largely indecisive and failed to really challenge the Crusaders - avoiding battle or fleeing after only token fighting. He eventually fled the capital, leaving Constantinople to the freed Isaac and Alexios IV (Isaac was a co-ruler but given his age and the effects of his blinding and imprisonment was unable to fulfill the requirements of the role). Unsurprisingly, the years of mismanagement made it impossible for them to pay the Crusaders what they had promised, even after various attempts to raise additional money, and tensions between the Crusaders and the Byzantines (who were understandably not pleased by the idea of giving all of their wealth to the Crusaders) continued to rise.

Eventually, tensions boiled over into armed conflict in 1204, but Alexios IV had already alienated too many of his own people and was himself overthrown by Alexios Doukas (a popular member of the nobility). Doukas attempted to fight the Crusaders more directly and vigorously and was somewhat successful for a time, but fled the city before the Crusaders had full control over it - he married the daughter of Alexios III but was eventually betrayed by him, blinded and handed back over to the Latin Empire which had him executed.

It wasn't just their ineffective (even sometimes self-destructive) leadership to blame, though. Even someone like Manuel Komnenos, who ruled from 1143-1180 and was seen as a great ruler at the time, played a role in the Byzantine decline by draining the Imperial treasury and being unable to fully deal with several major threats (particularly the Normans and Sejuks). Relations with others like the Venetians had declined significantly during his rule.

Though not his fault, he also left the Empire in a precarious position because his son (Alexios II Komnenos) was only 11 at the time of his father's death and the regency that ruled for him was unpopular. Andronikos Komnenos rebelled and overthrew his regency council, executing many of the young emperor's advisors. It was Andronikos that ordered the massacre of the Latins in 1182 that /u/Emperor_Pupienus mentioned in their post and who had Alexios II killed in 1183. Andronikos II was another unsuccessful and unpopular leader - unsurprising given his fate only two years later (he's the one that Isaac II rebelled against and overthrew, which kicked off this very post!)

So it was really a variety of factors - there was massive political instability starting from the very top, the emperors they did have were largely ineffectual or directly contributed to issues that weakened the Empire, there were several strong hostile powers that caused problems, previously friendly powers (like Venice) were alienated by things like the Massacre of the Latins, and the political and economic infrastructure of the Empire was simply not strong enough any more. You could also point to earlier events as a starting point for the substantive decline of the Empire - like the defeats against the Turks in the 11th century. The Empire did recover to some extent, as you note, but never fully.

As a starting point for reading material, I really enjoyed John Julius Norwich's trilogy on Byzantium (With this time period being covered in the third volume - The Decline and Fall)

I don’t think it’s fair to say “only Asia Minor and the Balkan Peninsula.” The implication here is that Asia Minor and the Balkans are naturally poor. That might be true enough for Southeast Europe at this time, but not necessarily for Anatolia.

Anatolia was poor at the time because it had been devastated and by war and revolt, and because political decentralization meant that the Byzantine state lost control of the manpower and resources there. But this was a consequence of a general decline in its ability to effectively govern from Constantinople.

Remember that Asia Minor was very productive in earlier Roman times, and in Hellenistic and even Achaemenid times. It isn’t a resource-starved or inhospitable place per se.