So any "why" question is difficult to answer. All the more so for "why not" questions. And in this case, I think it may be easier to answerwhy Christianity and Islam did get so big, rather than why Judaism didn't...but my expertise is in Judaism. After all, there are other small Abrahamic religions outside the "big three" (or big 2 plus small-and-famous 1). So my answer will focus on the things making the growth of Judaism slow and halting, but the assumed between-the-lines bit is that those factors did not exist for Christianity and Islam.
So in late antiquity, Judaism was a non-tiny religion, but a regional one. While there were Jewish communities around the Mediterranean and beyond, there weren't huge areas of a Jewish majority outside Judea. Individual cities had substantial proportions of Jews (Alexandria comes to mind primarily, but Babylonia had a huge Jewish community too, as did Arabia), but there weren't huge areas of Jewish dominance outside Judea (a situation not unlike today, actually. A Jewish dominated society in the Levant, and Jews spread out elsewhere, without regional dominance anywhere else, though with occasional local substantial presence).
A series of two major revolts in the 1st and 2nd centuries in Judea led to the decline of the Jewish community there, combined with general issues in the 3rd century with Roman governance. The diaspora, which had been an important part of Jewish life for centuries, became the based of Jewish population.
This is an important development. Not having a majority community means that conversion to Judaism was not expedient for anyone, really. No matter who conquered what, or who moved where, there wouldn't be an expedient reason to convert to Judaism. It never would be the socially preferred choice. It was the religion of neither the powerful elite nor of the common majority. Add to this constant restrictions and governmental prohibitions on conversion to Judaism, and it is apparent how Judaism's growth would be hampered.
(About those governmental restrictions, conversion to Judaism had had legal issues since Roman times, which continued intermittently. While conversion was an important part of the growth of Jewish communities, it took a different form from Christianity and Islam. Jews' neighbors might convert, or more likely their slaves. Jews might talk about Judaism and get people interested in converting that they knew. They weren't travelling to distant lands to convince them to convert.)
This was the case in both Christian and Muslim areas. Jews were a special legal category, who were subject to different laws. That sometimes included extra taxation, from 12th century England to 19th century Russia. Sometimes there was legally enforced separation. Other times there were massacres or forced conversions of Jews. A substantial chunk of the Jewish community of Spain converted forcibly to Christianity or died while fleeing--the exact proportion is not really estimateable. Russia attempted to assimilate Jews with a military draft, which doubtless succeeded in many cases.
There are two notable exceptions to that. There were two diasporic Jewish governments, the Himyari kingdom and the Khazars. The former were a group in the Arabian penninsula prior to Islam, when Judaism and Christianity were battling it out to be the Arabian monotheistic religion, with a backdrop of Arabian polytheistic religions. We know precious little about their rule from Jewish sources, and they didn't last terribly long before their dominant position was defeated by Christians and later Muslims. The Khazars converted to Judaism to some extent in the Middle Ages, perhaps to avoid Christian-Muslim conflict. But the extent and reasons behind their conversion are debatable. In this context, they are interesting mostly as exceptions--they're only two governments, separated by centuries, and neither managed to keep up their Jewish rule for particularly long, and in neither case did Jews elsewhere use this prestige as a way of converting people.
Later on, when Jews were more integrated, there were still powerful social forces keeping people from converting, and encouraging conversion out. This occurred in Eastern and Western Europe. When people saw benefits to converting to Christianity without the massive personal upheavals this would create (if you're integrated and assimilated, converting out won't change your entire life the way it would in earlier Jewish societies).
Sources:
I noticed most of the comments on this topic are meant to engage in debate with a comment that is now removed, leaving this question with 29 comments and no decent attempts at an answer. Could one of the people who replied to the original responder write an answer to OPs question?