Even at the time of the first crusade there was some degree of mistrust between the westerners and the Byzantines, but the most important event between 1095 and 1204 which alienated the westerners was the Massacre of the Latins. This was a mob-led massacre of Catholic merchants living in Constantinople which led to tens of thousands of deaths.

For some background, prior to the massacre the Empire was led by Manuel I Komnenos. Manuel I was sort of unusual--he wasn't exactly pro-Catholic or anything, but he was married to a Frankish woman and introduced some western customs to the Empire (for example, he held western-style tournaments with jousting and stuff like that, which was not a typical Byzantine thing). When Manuel died, his Frankish wife Maria of Antioch took over as regent for their underage son, Alexios II. She was resented not only for being a foreigner, but because she showed favoritism to other foreigners such as the Pisan and Genoese merchants who dominated trade in Constantinople at the time. Her regency was overthrown by Andronikos Komnenos, a relative of the Emperor who took advantage of the anti-foreigner sentiment to sieze power. He had Maria killed and initially became "co-emperor" with Alexios II... who Andronikos soon had deposed and killed as well.

The anti-foreigner catharsis of deposing the widely disliked Maria of Antioch boiled over into mob violence against the western merchants. Andronikos allowed his supporters to go ahead and do whatever they wanted to the Latins (overall Andronikos was an inept and cruel leader, who eventually died to mob violence himself). About 60,000 westerners lived in Constantinople at the time, and a huge number of them were killed in the attacks. From Enrico Dandolo and the Rise of Venice--

Andronicus signaled the citizens that they might give full vent to their hatred for their Latin neighbors. The massacre that ensued was brutally efficient. The Greek mobs poured into the Latin quarters along the Golden Horn, murdering, raping, and torturing their victims. The easiest targets were women, children and the aged, who were cut down mercilessly. Latin clerics and monks were also massacred; the papal legate to Constantinople was decapitated and his head tied to the tail of a dog. The slaughter so embarrassed Nicetas Choniates that he simply omitted it from his history.