Jews being mostly in Europe is a Middle Ages phenomenon. In the early Middle Ages, there were large Jewish communities in Europe already--Spain and Italy, mostly, with smaller ones in other parts of Europe. But these were not yet as large as Jewish communities in the Middle East, such as Egypt and Babylonia (though in the case of Egypt, it's worth nothing that both Egypt and the large European communities of the time were mostly Mediterranean).

Over time the European Jewish communities expanded into other parts of Europe--first France and Germany in the Early Middle Ages, then Eastern Europe. Middle Eastern and African Jewish communities didn't really spread out that way, at least not to the same extent. Intuitively I'd think that increased geographic area for Jewish communities to exist would result in population increasing as there's space, but that's a conclusion I'll leave to someone who knows more about demography than I do.

But the most important bit is actually early modern, and that is birth rates. The European Jewish community grew extremely fast during the late Middle Ages and early modern period. This growth rate was what has led people to suspect the Jews involved came from some other population source, such as the Khazars. But there are several other possible explanations. Documentary evidence suggests Jews in Europe in the Early Modern period tended to marry much younger than their non-Jewish neighbors. Until the mid-1800s most Jews in Eastern Europe married by 20. In 1797 the Latvian census showed that about half of Jewish women from 15-19 were married, but only 9% of Christian women in that age bracket were.

The problem here is that we don't have very good documentation of people in general from this era, and when we do have it it's not necessarily very good. Jewish communities tended to not register marriages with the government, and often didn't bother registering births either (which meant that many people were registered as illegitimate, when their parents were married, but the marriage had only occurred within the bounds of the community, and no one had seen fit to involve the government). So the picture is incomplete. But there's plenty of other evidence for young marriages. It seems that marriage in the mid teens was not terribly uncommon.

Young marriage like that results in two important things. First, the earlier people get married the more child-bearing years they're married, which means they'll probably have more children. But more subtly, it means population growth from one generation to the next happens more quickly. If every couple has 3 children born to them at 18, each of whom goes on to do the same, the population would grow far faster than if they had those children at 25.n

The answer is different in different parts of Europe. The Italian Jewish community is very old, and most likely came straight from the land of Israel. Jews have lived there since around the second century BCE.

There are still debates about where the Jewish community in Germany and France came from, but the current most popular hypothesis is that they came from Italy.

Related question: Is the Khazar theory completely discredited in academic circles or is more like "We can never know for sure, but could be true, just unprovable."?

In addition to what others have said, there were a couple of way Jews go to Italy during Roman times. First, was through trade and economic relationships. While most Jews lived in Judea, others would settle in major cities of the Empire, including Alexandria and Rome to pursue economic relationships. Second, was that after the destruction of Jerusalem by Titus, many Jews were taken back to Rome and Italy by force (Josephus is a prominent example). Over time, Jews would move North (in accordance with the moving of the capital from Rome to Milan to Trier to Ravenna (modern names)) and many ended up in the Diocese of Italia Annonaria, which makes up parts of Northern Italy, Switzerland, and Southern Germany, and Diocese Gallia, which makes up parts of Eastern France and Western Germany. Notably, Jews of the latter formed the origin of the "Shum" communities in Speyer, Worms, and Mainz, as well as other Jewish communities in southern Germany. These Jewish communities would then form the majority of Jews who ended up in Eastern Europe as events like the massacres during the Crusades and the tolerant policy of Boleslaw III (later notably held up by Casimir III) prompted Jews to move East.

Some Spanish Jews ended up in Eastern Europe, but it was mostly these German Jews who had their origins in the Roman Empire who populated Eastern Europe.