I've put the dialogue in question below for context, which takes place at the 50th anniversary dinner of Jerry's relatives Manya and Isaac. I've bolded the quote from Manya that implies that pony ownership was somewhat common, at least in her village. Furthermore, Larry David, the Seinfeld creator and the writer of this episode, has said that the episode was based on a real remark he made. David is of Polish Jewish heritage on his mother's side.
HELEN: (Changing subject) So, did you hear Claire's getting married?
MANYA: Yeah, yeah..
HELEN: I hear the fella owns a couple of racehorses. You know, trotters, like at Yonkers.
JERRY: Horses. They're like big riding dogs.
ELAINE: What about ponies? What kind of abnormal animal is that? And those kids who had their own ponies..
JERRY: I know, I hated those kids. In fact, I hate anyone that ever had a pony when they were growing up.
MANYA: ..I had a pony.
(The room is dead quiet)
JERRY: ..Well, I didn't really mean a pony, per se.
MANYA: (Angry) When I was a little girl in Poland, we all had ponies. My sister had pony, my cousin had pony... So, what's wrong with that?
JERRY: Nothing. Nothing at all. I was just merely expressing..
HELEN: Should we have coffee? Who's having coffee?
MANYA: He was a beautiful pony! And I loved him.
JERRY: Well, I'm sure you did. Who wouldn't love a pony? Who wouldn't love a person that had a pony?
MANYA: You! You said so!
JERRY: No, see, we didn't have ponies. I'm sure at the time in Poland, they were very common. They were probably like compact cars..
MANYA: That's it! I've had enough! (She leaves the room)
ISAAC: Have your coffee, everyone. She's a little upset. It's been an emotional day.
(Isaac leaves, everyone looks at Jerry)
JERRY: I didn't know she had a pony. How was I to know she had a pony? Who figures an immigrant's going to have a pony? Do you know what the odds are on that? I mean, in all the pictures I saw of immigrants on boats coming into New York harbor, I never saw one of them sitting on a pony. Why would anybody come here if they had a pony? Who leaves a country packed with ponies to come to a non-pony country? It doesn't make sense.. am I wrong?
I really want to help you out here so I will do the best I can, especially given my limited knowledge of ponies vs. horses.
Records indicate that during the second half of the 19th century something like 3% of Jews worked in the field of transportation: coachmen, carters, porters, etc. Jews were pretty dominant in the field; by the end of the 19th century, about 21% of independent coachmen and carters were Jewish. This means that quite a lot of Jewish people in Poland (among other areas in Eastern Europe) would have owned a horse for work. This figure doesn't include others who, though they did not work in transportation, would have had a horse for work (for example, Tevye from Sholem Aleichem's "Tevye the Dairyman" has a horse that features prominently in the stories. He's a milkman, and he travels with a horse-drawn cart to deliver the milk). The nature of Jewish involvement in transportation would change during the 20th century, as transportation became increasingly mechanized. Nonetheless, horses would continue to be used for work and transportation, especially in smaller towns and villages (if you look up "horses" on the YIVO website, you'll find a bunch of pictures from market days and other moments capturing regular prewar life with dates up to the 1930s).
You'll note that I've only mentioned horses, not ponies, and horses that were working horses and not, presumably, someone's pet.
I'm going to make a series of educated assumptions about Manya in order to determine how unusual it would have been for her, her sister, and her cousin (and others) to all have ponies. First, I assume that the character is approximately 81, the same age as the actress Rozsika Halmos was when she played her. The episode aired in 1991, which means she was born around 1910. Because there is no mention of Manya being a Holocaust survivor, and because she has an accent, I'm also going to assume she came to the US at the last moment, say 1938 when she would have been 28 years old. This means that she would have missed the presumed peak of Jewish horse ownership in the late 19th century. This does not mean that her family would not have owned a horse, but it does make it more likely that owning not just one, but at least two horses would have been more unnecessary, and therefore more unusual for the time period.
You'll notice again that I keep referring to horses, not ponies. I didn't find photos of ponies the way I found some of horses in the YIVO archives. And based on my cursory search to figure out what you would use a pony for in general, I have to conclude that it wouldn't have made much sense to have a pony helping with your work when you had horses available. None of this means that there were no ponies in prewar Poland, just that it does not seem to have been at all common for people to have had them.
Conclusion: Manya's family and village would have been very unusual in having so many ponies. Owning a horse would have been more common (though less common in the 20th century than it would have been in the 19th), but even if we're talking about horses it would still have been highly unusual for it to be Manya's horse or her sister's horse, or her cousin's horse as opposed to a horse owned and used by her father for his work.
Edit: After doing some more research about horses in Poland!, I take back my statement about how rare ponies must have been, and how it was unlikely they were used for work. My original answer as stated in the conclusion still stands, but it is more possible than I originally thought that Manya's family would have had a pony.
Sources: Mostly https://yivoencyclopedia.org/article.aspx/Transportation. I recommend also searching "horse" on the site; you'll get some neat pictures.
There's also lots of Yiddish literature that features men (it's almost always men) and their relationship with nature/their animals. I mentioned Sholem Aleichem's Tevye the Dairyman, but I would also recommend checking out Mendele Moykher Sforim's work.