I know that, after the 3rd Jewish war, the Romans banned Jews and Christians from living in what they then called Palestine. Presumably this rule was overturned at some point. I know that some Jews lived there in the Middle Ages, so presumably some had gone back. But why were most Jews still living in other places? Of course, you can argue that they didn't want to move, having found new homes, but as they faced persecution at various points, having to move around to different countries, why not go back to Israel? Especially as the Caliphate was, generally-speaking, more tolerant than Christian Europe.
I'm not an expert on Zionism but I do tangentially read about it a good bit and I think the most succinct answer that I could give would be that Zionism in the sense of seeking to establish a Jewish nation-state in the historical territory of the ancient kingdom of Israel simply requires multiple levels of abstract Jewish national identity that simply did not exist before modern times.
As a form of nationalism, Zionism posits 1. that the Jewish people are a nation, 2. That they are a nation in need of a state, 3. that that state ought to be established in what by then was known as Palestine.
A nation, in the words of Benedict Anderson, is an 'imagined community' such that even if we stipulate, through evidence such as Jewish Ghettoization, that the Jews thought themselves to be a distinct community in the middle ages (point 1.) we know factually that points 2. and 3. of the Zionist program simply didn't exist until much, much later.
So for instance, when you say:
Especially as the Caliphate was, generally-speaking, more tolerant than Christian Europe.
Sure, Islamdom was often more tolerant of Jews, generally speaking, than many parts of Christian Europe. But Jews who migrated to Islamdom (like those who were expelled from Spain after the reconquista and who went to the Ottoman empire), did not therefore establish a program of creating a new state, nor did they wish to settle in Palestine despite Palestine being within the territorial borders of the empire in which they now resided. Unsurprisingly they used that tolerance to try to prosper in imperial centers like Istanbul rather than a barren imperial periphery like Palestine.
An account must also be made of the long tradition of Jewish non-Zionism or anti-Zionism. Even after the publication of Herzl's Der Judenstaat, which is often taken as the birth of the Zionist movement (despite significant precursors, including even among gentiles) a large portion and perhaps even a large majority of, for example, American Jewry rejected Herzlian Zionism. "America is our Zion" the Union of American Hebrew Congregations declared three years after Herzl published Der Judenstaat.
Even the document that was arguably most significant in actually establishing a Jewish state in Palestine, the Balfour Declaration, met significant resistance from Jews in the British government on the basis that such a declaration would undermine the rights and status of Jews as British citizens (or citizens of other countries).
It's not actually until the Balfour declaration is an established fact that American Jewry, for instance, gains majority support for a Zionism in Palestine (which remains purely a political support, as only a small fraction of American Jews ever actually migrate), and certain portions of Jews (e.g. Reform Jews and portions of the ultra-orthodox) remained opposed.
Sourcewise, for the American portions (which I'm most familiar with as opposed to international Jewish attitudes towards Zionism) see The Columbia History of Jews and Judaism in America and Howard Sachar's A History of the Jews in America.
On Jewish opposition to the Balfour Declaration within the British Government, virtually any history of the subject will cover it, but see for example Lord Montagu's memorandum on the subject here.