I can't speak as confidently from the Christianity side as I can from the Jewish side, but ancient Judaism had several factors running against it. First and foremost, Judaism had several practices that the other cultures of the Mediterranean considered barbaric and alien. For example, Strabo, in his work the Geographica, describes Jewish history, culture, and religion in some detail. He is critical of their traditions, going as far as to label their circumcision as koloboi tas balanous, or mutilation of the sexual glands. Apion and other scholars of Alexandria close to the dawn of the common era were especially critical of Jewish tradition, and their works had a wide audience, including the Roman emperors. Even simple ideas, such as Jewish eating habits, seemed very strange and superfluous to others at that time. Josephus claims that these opinions were born of a mix of ignorance and ill will -- and through these works, the Jews assumed a label of barbarians according to Greco-Roman scholars.

Additionally, after the Jewish-Roman revolts, the Romans began a systematic destruction of Jewish social structure in Judea and elsewhere, leading to the modern diaspora. This can be labelled as one of the definitive starting points for heightened Antisemitism in the ancient Mediterranean, and Roman efforts to demonize the Jews after these conflicts would also factor against the spread of Judaism.

Christianity, on the other hand, did not continue these traditions, especially after the Council of Jerusalem, meaning the religion became more appealing to those who did not favor the other facets of ancient Judaism.

Tessa Rajak's work The Jewish Dialogue with Greece and Rome; Studies in Cultural and Social Interaction provides some essays that explore the relationship between the Greco-Romans and the Jews. The ancient sources themselves, Josephus primarily, also explores this idea. I'm not as well versed in the Christian side of things so I'd leave that to another, more venerable historian.