Excellent question! The answer is "the whole thing is a lot more recent than people think."

Chelm is, of course, a real place. It's now a Polish town with a bit over 60,000 inhabitants, and before WWII it had a large and robust Jewish community, though before the 1648/49 massacres it had been of much greater importance and dignity (a prominent yeshiva was based there as was Rabbi Elijah Baal Shem, a mystic who, according to folklore, is supposed to have made a golem). It was a perfectly ordinary community, and yet, sometime in the mid-19th century (the most precise date we can find for the first print reference is either 1867 or 1871), it became associated with Jewish tales of foolishness, most notably with the phrase "wise men of Chelm" (in Yiddish "Chelmer chachamim").

Now, this isn't to say that the tradition of "village idiots" and jesters and such was new to Jews by the time Chelm as such became a thing- while they had a lot in common with German archetypes, Jewish versions existed for centuries before the first Chelm reference was made. Often these depictions centered around the holiday of Purim, known as a time of merrymaking, but not exclusively- and of course the long-standing tradition of the wedding badchan, or comedian, fits well into these traditions. Another Jewish jokester figure in stories, Hershele of Ostropol, was likely partly based on the German prankster figure Till Eulenspiegel but also acquired his own very Jewish flavor. There is also a fool character called Choizek who was used in some stories, including later Chelm stories (I'm more familiar with the term "making choizek" which is used in Yinglish to mean "making fun" or "making a joke").

The first known Yiddish-language stories about foolish people who lived in a small town were published in 1598, but they weren't about Chelm, or even original- they were a Yiddish translation of a recent German book called the Schildenburgerbuch, stories about the German town of Schilda, which was given this reputation as being a town of fools. Some Chelm stories can directly trace their roots back to stories about Schilda. The stories became very popular among Jews in the 18th century, and multiple Yiddish editions were published. But Jews developed, in parallel, many of their own stories and conceptions of what a village of idiots might mean, as recorded by Abraham Tendlau in 1860. He collected Jewish stories and folklore, among them references to the Jews of Prague, Metz and Furth as being particularly lacking in common sense (as well as the Jews of Frankfurt being egotistical and the Jews of Worms having experienced miracles). Other stories referred to the Jews of Posen, as well, as being known to be foolish. Bringing the two different traditions- Schildenburgerbuch and Jews' on folklore- occurred at the end of the 19th century, as we've seen. By 1873 the term "Chelmer narrunim" (or "fools of Chelm") was first in use in print, specifically ascribing to Chelm the characteristic of having foolish inhabitants, and by 1892 we see Chelm listed as "known for foolishness" alongside two other towns, Lesko and Nadvorna, with Chelm described as "most famous." This last reference, by Binjamin Segel, is the one that cemented Chelm, and Chelm stories, into the public consciousness- it was not in a Yiddish book but in a German-language journal. Other later articles assigned foolishness status to other towns as well, but Chelm is the one that lived on.

It's at this point that the "canon" of Chelm stories first began to crystallize in writing- and by the 20th century the idea of the Chelm story- whether adapted from folklore or created by the author (such as one which was intended by Segel, the author, to be an analogy to the plight of Jews during WWI)- became a standard in Yiddish literature. By 1917 it was reported that hundreds were in circulation, some in the Schildenburger tradition and some written by new authors, including such dignitaries of Yiddish literature as Sholem Aleichem, IL Peretz, and later Isaac Bashevis Singer- whose Chelm stories are younger than my parents. It raises a question- should these qualify as folklore? It's hard to say, but definitely as part of a folkloric tradition. As Chelm stories have become popular fodder for media, particularly children's books, they are further entrenched

So- why Chelm? Why is an ordinary city slandered in such a way? It's not super clear. As we've seen, Chelm is only one of the Jewish communities that has been called foolish over time, but it's interesting to ponder why it specifically stuck. One scholar has said that it's because the phrase Chelmer chachomim, a sarcastic reference to the "wise men of Chelm," became popular due to its alliteration, or that Chelm and "chalom/cholem" (Hebrew/Yiddish for "dream") sound so alike. Chelm was also known as a bastion of traditional Judaism, particularly the odd-looking Chassidic Judaism, leading Jews who were part of the secularizing Enlightenment to perhaps cast them as foolish. Or maybe it was the result of a rivalry with another town. But it happened, even if it turns out it only happened by chance, and now Sacha Baron Cohen is apparently making a show about it on HBO Max. Rabbi Elijah Baal Shem would probably find this weird.