When the Roman Empire converted to Christianity, why did only those who worshipped the Pantheon get converted, but not the Jews? Shouldn't Christianity have been more popular in Israel, where it originated, than in Europe, as is the case with just about every other religion?
I'm not sure why you'd say that it's more usual for a religion to be more popular where it originated. There are hundreds of millions of Buddhists out there...and only a relative handful in India.
Jews traditionally had a very different idea of what a Messiah should be. A great king, from the line of David, annointed by God to crush the enemies of the Jews, bring all the Jews back to the Holy Land, and usher in an era of peace and prosperity.
Needless to say, this bears little resemblance to Jesus' career arc, thus leading to heavy Jewish skepticism about the claim he was the Messiah.
It's important to remember two things when looking at this question.
Is that Greek paganism was not the religion of the Roman Empire. The Romans had their own pantheon of Gods, as well as different religious practices to the Greeks.
Judaism is not an Evangelical religion, and by this I mean that unlike Christianity and Islam, Judaism does not actively convert people. To be Jewish, one's mother has to be Jewish. It is not a case of being baptised, or saying that you have converted. You actually have to be born Jewish.
I can give you one perspective among many, although whether we can ever completely know the answer to your question is debatable.
One factor leading to the unpopularity of Christianity among Palestinian Jews, a factor proposed by Magnus Zetterholm in The Formation of Christianity in Antioch, may have been that "normative" Palestinian Judaism and nascent Christianity (in the first century, perhaps still best defined as a sect of Judaism) very quickly found themselves fighting over the same identity. Because in a very important sense they claimed to be the same thing, they were forced to differentiate; through this process, any hope of reconciliation was lost.
I will try to explain what I mean. Due to the antiquity of the Jewish religion (it predated the Roman state by a large margin) as well as its tenacious monotheism (Judaism forbade the worship of other gods, whereas many indigenous religions could merely adopt new gods in addition to the old if the region was conquered), Jews were granted a special exemption from any requirement to participate in the imperial cult. This allowed for peace between Rome and the Jews. This peace was shaky as evidenced by the near catastrophe when Caligula ordered a statue of himself to be erected in the Jewish Temple. But the exemption and consequent peace held for a number of years.
The Jews were identified as the descendants of Abraham, Isaac, and Jacob; as another commenter has pointed out, the Jewish religion cannot really be separated from Jewish ethnic/cultural/"national" identity. The Christians, for the most part, were not descendants of the Patriarchs; although some Jews followed Jesus, the number of Gentiles soon grew to outnumber the Jews in the Jesus movement.
The Christians adopted Jewish laws regarding idolatry in essentially unchanged form. Because of these various prohibitions, they could not burn incense to Caesar or worship the gods of the Roman pantheon or even their ancestral gods. To leave off worship of these gods and especially of Caesar was an act that lies somewhere in between socially stigmatized behavior and outright civil disobedience, depending on who you read.
So how did the Christians propose to solve this problem of being, essentially, an illegal movement? --Claim to be the TRUE descendants of Abraham, Isaac, and Jacob, and thereby fall under the umbrella of the Jewish exemption which allowed all Jews to refuse participation in the imperial cult.
So now we have two groups claiming to be "Israel," to be the true form of the ancient religion that deserved the exemption initially granted to the Jewish people.
As you can imagine, the Jews were fearful of losing this exemption, and maintained their claim over against that of the Christians.
Zetterholm's thesis is that this struggle over identity, over who really got to call themselves children of Abraham, was a critical factor in the separation of Judaism and Christianity. Because Jews and Christians claimed the same identity, and because their claims were perceived to be mutually exclusive, these two groups highlighted and magnified their differences in order to differentiate themselves. To put it another way, these two movements developed a tendency to define themselves against each other. Christianity was not Judaism; Judaism was not Christianity.
To give some specific examples, Jews would see a Jewish tradition that had been adopted by Christians--the recitation of the Ten Commandments being one historically supported example--and deprecate it. Christians likewise stripped some of the more "Jewish" elements from their own practice; the current location of Easter on the Christian calendar is the result of a deliberate effort to disconnect it from the date of Passover, so as to eliminate Christian dependence on Judaism. A major driving force behind these changes was to establish a concrete identity through differentiation. You can see this same phenomenon in recent history through the proliferation of Protestant denominations, exploited to great comedic effect by Emo Philips. https://www.youtube.com/watch?v=BDmeqSzvIFs
Because these two movements so quickly defined themselves in opposition to one another, by the time Christianity went mainstream a few centuries later, the Jews (as a group; there are always exceptions to statements like this) were not interested, and had not been interested in some time. From a social rather than a religious perspective, we could say that each group felt betrayed, in some sense, by the other, when each maintained its identity claim to the Roman government. (This sense of betrayal may even be the source of anti-Jewish polemic in some of the later writings of the New Testament. Revelation 3:9 for example.)
So it happened that two religions born from Second Temple Judaism found themselves to be so diametrically opposed, despite their shared origin and enormous similarities. That is, if you accept Zetterholm's thesis.