Great question!
Unfortunately we don't really know. While the presence of Jewish communities all around the Mediterranean are documented, the actual hard information on their populations is scant. And more importantly for your question, the sort of narrative of why people migrated the way they did requires a level of documentation we just don't have. We can imagine that some were sent to each place by the Romans as slaves, but (contrary to the popular narrative) there were Jewish communities in much of the Roman Empire before then, and much of the out-migration seems to have been later. The most likely reason for many of them was simply economic, but there's no way to figure out the details of how many were migrating why (though I'm hoping to track down some better sources for this). We do know Jews were working all around the Med in all sorts of jobs, including the imperial bureaucracy. Some ancient historians reference overpopulation in Judea, which would've provided the economic push to go elsewhere. One interesting one we do know about are Jewish soldiers in the army of the Ptolemaic Empire who settled in Egypt.
The basic point is that "the diaspora" is a gradual process where Jewish communities ended up all over Europe and the Middle East, rather than an individual moment when people were taken in large numbers from Judea to other places. While there undoubtedly were forcible dispersions of Jews by the Romans during wars, that's not the only reason why Jews ended up in other places. Real dispersion--where everyone, or nearly everyone, was killed or enslaved and exiled--only happened in Jerusalem.
Mizrahi Jews come from a much broader area than Israel. Definitions get a bit sticky here, and Mizrahi is an awkward label for people who often don't identify that way (I'm told that Syrian Jews often much prefer being called "Sefardic"), and whether or not Yemenite and North African Jews should be counted as "mizrahi" (the former because they very much have their own independent liturgical, textual, and cultural traditions; the latter because they aren't east of anything and also kind of have their own identity--and many North African Jews are descended from Spanish Sefardim). So the existence of Jews all around the Middle East doesn't really prove that people could've stayed in Judea from Roman times onwards. In fact, prior to the migration of Jews to Israel as part of the Zionist project to bring about Jewish statehood, the population of Jews there was quite diverse. There were some people who might be called Mizrahim, part of the world of Middle Eastern Jewry. After all, administratively the region was a subsection of Syria, which had a very large Jewish community. There were Sefardim who had migrated around the Med after the Spanish expulsion. And there were Ashkenazim, who'd migrated in the 1700s.
But all that sort of point out why the population in Palestine had not been stable from the Roman period onwards. The region had times of economic trouble, and eventually the Middle Eastern community that came to dominate it (in terms of religious and cultural prestige) was Babylonian Jewry, showing the decline of Palestinian Jewry. The Crusades caused serious disruption to Jewish life in Palestine, with whole communities being displaced or killed. While the Jewish population eventually did bounce back from migration from other areas, it kept the population relatively small.
Sources: