Their strict adherence to monotheism made polytheists discriminate them. Then Christianity came, the anti-semetism the European churches preached and spread throughout the world via conquest and missionaries also made things worse. Biblical examples like the expulsion of Jews by Babylon and the one you gave of Haman aren't really anti-semetic in my opinion. Those things happened because of political factors, not just because the Jews were Jewish.