So there are two different elements at play here- lack of growth and active suppression. Both are, obviously, linked.

Back in the day (and by "the day" I mean "1500 or more years ago") Judaism was more open to proselytization than it is today. The thing is, Judaism never saw proselytization as a fundamental part of its mission or theology even when it was more open to or encouraging of converts. For example, even in ancient Rome, when proselytization was more of a phenomenon, there was also a perfectly well accepted phenomenon of God Fearers, non-Jewish Romans who were interested in Judaism and maintained affiliations with their local Jewish communities without ever converting to Judaism. So even if Judaism were to grow through conversion, it would not necessarily be in the kinds of exponential numbers that Christianity and Islam, which did see conversion of non-Christians/Muslims as part of their missions, achieved.

If proselytization is pretty much taboo in Judaism now, and conversion is so restricted and rare, a not insignificant part of that is due to the fact that for a millennium and a half, in many of the places were Jews lived, they were banned from converting people to their faith. Once Rome became Christian, for example, not only were people banned from converting to Judaism, but even the God Fearers (who were, again, never actually converted to Judaism) were banned. Of course, you don't ban something that isn't happening, and there was still some conversion to Judaism, by pagans and Christians, at this point, but still- once both Christianity and Islam became dominant forces in the places where Jews lived, they tended to ban conversion to religions that weren't, respectively, Christianity or Islam, which limited the non-natural population growth of Jewish communities.

And here we get to the difference, yet link, between lack of active growth and suppression. Because on the one hand, preventing active growth is a form of suppression- but on the other hand, there was also active, even violent, suppression. After all, and in the case of both Islam and Christianity, mass conversions to those religions often meant that Jews were, themselves, being converted, often violently (depending on the time and place). To get closer to the time periods I usually feel comfortable discussing, for example, in medieval Europe during the Crusades there were many instances in which Jews were told to "convert or die"- and, indeed, often chose to die, though not always- and in Almohad-ruled Spain and North Africa, many Jews nominally converted to Islam while secretly practicing Judaism (including, perhaps, the great rabbi Maimonides himself!).

Then, of course, you get to the ultimate form of suppression, alluded to earlier, which is actual murder, which was also definitely a factor at this time. The early years of the Crusades were particularly deadly to Jewish communities in France and the Rhineland, and a few centuries later the pogroms that came alongside the Black Death were also extremely destructive. In fact, these were SO destructive that geneticists believe that one of the reasons why the Ashkenazi Jewish gene pool is so small, alongside endogamous marriage, is just that it was small to start with due to the murder of so many of the original community members.

Community repression, antisemitism, and expulsion were also deeply problematic when it came to communal growth, and also often came deeply tied in with murder. For example, let's break down the situation in Spain- the situation is already shaky for the Jewish community when, in 1391, there are pogroms which break out across Iberia, leading to many deaths and many conversions by those who wished to preserve their lives. Those Jews who did not convert found themselves suppressed in terms of opportunities and lifestyle and were constantly encountering efforts to convert them, such as forced lectures in their synagogues. 80-some-odd years later, the Inquisition is established to prevent Jews from being able to maintain their religion secretly while outwardly seeming like Christians, and then 100 years after those pogroms, the remaining Spanish Jews were told to convert or leave. Those who converted faced the Inquisition if they retained their Jewish faith; those who left had to leave nearly all their possessions behind and faced death in the form of disease, shipwreck, piracy, and more, with untold thousands dying on their journey from Spain.

While this is all a particularly large and notable instance, various elements were in common in many places where Jews lived throughout the medieval and early modern eras. Expulsion could cost many Jews their lives, especially if it came in the wake of pogroms. Many places where Jews lived only allowed them to stay under the restrictions of oppressive charters, which limited the number of families allowed in the towns, demanded massive amounts of money in return, and allowed Jews to be expelled on a whim. This is one of the reasons why Jews in central Europe, where communities were small and restricted and where expulsion was common, made their way in the late medieval/early modern era to Poland, where the king was welcoming and restrictions were fewer. Suppression and poverty among Jews could lead to population shrinkage, whether in Christian or Muslim lands, and in fact in the early modern/modern era Jewish populations in Islamic lands began to shrink greatly for these reasons.

With the modern era, and increased quality of life for Jews in modernizing societies which gave them equal rights (as well as, shockingly, the arrival of the potato!), Jewish populations began to expand again. Emigration to the US led to its Jewish population growing exponentially in the 40 years between 1881 (when pogroms in Russia following the Tsar's assassination led to huge waves of Jewish emigration) and the early 1920s (when anti-immigration laws were passed in the US)- millions of Jews arrived and were generally given the opportunity to thrive, leading to Jewish populations in the US which, while still a small fraction of the country as a whole, are still numerically large. Jewish populations in Europe grew as well despite the emigration to the US, until the Holocaust wiped out a third of the world's Jewish population. 75 years later, in a world in which Jews by and large have places to live, including a Jewish state, without significant limitations and in which the world population at large has been growing in leaps and bounds, the Jewish population has still not quite recovered to its prewar peak of about 17 million.