As a caveat, I'm a music theorist, not a music historian, so my answer will have a strong theoretical bent, with little discussion of actual music.

I'll begin by clarifying some terminology. Strictly speaking, time signatures don't indicate rhythm. The notation inside the time signature indicates rhythm, which is just the specific notes that happen to arrive at various points in time. Rather, the time signatures indicate meter. Meter is a structure lying underneath rhythm, entailing when the strong beats are, how often the strong beats recur, and so forth. Any meter has numerous possible rhythms. Some rhythms might have several possible meters---is the opening rhythm of Steve Reich's Clapping Music in 3 or 4 beats?

There are two ways to interpret your question, then. On one hand, we could ask "why are these time signatures the most common in notation and theory?" On the other hand, we could ask "why does most Western music follow these time signatures?"

The first one is more directly historical. Time signatures developed in the 17th century out of an earlier system of "mensuration signs" (Caplin, 658-59). These signs didn't necessarily indicate meter or an aspect of rhythm, but rather a "theoretical grid that serves as the system of reference for the note symbols" (Deford, 2). (The note symbols back then resembled our modern quarter- and half-notes, but weren't exactly the same.)

Within the system of mensuration, there were multiple levels---smaller notes are subdivisions of larger notes. There were four symbols indicating how two of these levels in particular were divided---either into two or three parts. For example, one "measure" (to use an anachronistic, but clear, term; the proper term would be "breve," which is a certain note length) could be divided into 2 parts, which could in turn be divided into 3 parts each. (For more details on this system, see Deford, 33-39.) There were then four different symbols, corresponding to whether the longest notes were divided into two or three parts, and then whether the next notes were divided into two or three parts.

So when time signatures developed from mensuration, there were a highly limited number of possibilities---a time signature would be unlikely to not follow one of the four possible divisions of twos and threes. 4/4 and 2/4 come from dividing in two and two, 3/4 from three and two, 6/8 from two and three, and 9/8 from three and three.

One could ask, of course, why we have two and three as the basic units in mensuration. I'm honestly not sure anyone would have a good answer for that---the first known Western rhythmic notation already divided things into six parts, consisting of two larger parts each divided into 3 smaller parts (Busse Burger, 628-630).

So that provides an answer about theory. But what about the music? Why should music follow these signatures? One argument would be that music hadn't always followed these signatures, and when music was written down with rhythm, it might just have followed the notational system available. Such constraints might seem ludicrous to us now, but are well-documented in how chant was first written down according to preexisting Ancient Greek and Roman theory, simply because it was the tool available for describing music (Atkinson, 158-170). And in terms of rhythms, there was certainly a lot of selectivity in what music was written down. A lot of the more complex music of the time would have been composed with the mensuration in mind.

For the sake of completeness, I'll also provide a non-historical answer that many might give for why "weird time signatures aren't common:" cognitive constraints. Justin London (2004) presents a theory of meter in which he derives most common meters from a set of rules based in cognitive research; divisions into 2 and 3 are essential, and although he does come up with justifications and constraints for 5/4 or 7/8 and the like, they require him to revisit his initial premises. One might speculate that the basic and consistent divisions into 2 and 3 being in common between medieval-renaissance mensuration theory and modern cognitive theory isn't a coincidence, but that's not a leap I'd particularly want to take.

Sources:

Atkinson, Charles M. The critical nexus: tone-system, mode, and notation in early medieval music. Oxford University Press, 2009.

Busse-Berger, Anna Maria. "The Evolution of Rhythmic Notation." In The Cambridge History of Western Music Theory edited by Thomas Christensen, 628-56. Cambridge University Press, 2002.

Caplin, William E. "Theories of musical rhythm in the eighteenth and nineteenth centuries." In Cambridge History of Western Music Theory edited by Thomas Christensen, 657-694. Cambridge University Press, 2002.

DeFord, Ruth I. Tactus, Mensuration and Rhythm in Renaissance Music. Cambridge University Press, 2015.

London, Justin. Hearing in time: Psychological aspects of musical meter. Oxford University Press, 2012.