There is definitely a physical reason for this. The harmonica makes music through the vibration of a reed. There are two reeds per hole, one for when you blow and one for when you draw (er, suck). The vibration of the reed, along with hand shape on the back of the instrument and even the shape of your mouth/tongue can affect the character of the sound. Unlike the clarinet or saxophone, the reed is not vibrating in between your lips, but instead it is part of a larger reed plate and vibrates in place. If you practice the draw quite a bit, you can also cause the reed to bend in place and drop several semi-tones to change the note itself.

In a saxophone or clarinet, the reed sits partially in your mouth and is attached to a plastic mouth piece. Air pressure and lip pressure make a buzzing sound that makes the noise. Inhaling will not work just because of the structure of the instrument itself. As for the brass instruments (tuba, trumpet, trombone), they make noise by "buzzing" your lips into the mouthpiece. Think of making a raspberry "thpppppt" sound and that's what buzzing is. It's not possible to do that inwards. Also the sound of brass instruments is affected by the length of tubing the air passes through. If you straightened all the tubing of a trumpet it would be much shorter than that of a tuba.

For recorders and flutes, noise is made by blowing air across an open hole and then adjusting air escape by opening or closing holes. Inhaling may work on a flute, but the tones would probably be the same and wouldn't sound as clear as a good solid, controlled exhale through pursed lips.

Overall, it's just based on how the instruments are made and how they make noise.

For some context: the harmonica is a free-reed aerophone, which means that it depends on a series of tuned reeds, each of which is intended to sound a specific pitch when air causes it to vibrate. You blow, you get one set of tones. You suck, you get a different set. The only other truly comparable Western instrument is the accordion - where (usually) air travels in two directions, (usually) activating a different set of reeds each time, producing (usually) different tones. The accordion, being larger, requires powering via a bellows-like mechanism, and the air moves across the reeds as you pump in and out with your arms. The other aerophones you listed - flutes, trombones, saxophones, etc - while they have a variety of mechanisms of action, they are all different from the harmonica and accordion in that they achieve variation in tone by, simply put, changing the qualities of a single air pathway, as opposed to activating one of a series of individual, separate air pathways.

So the simple answer to your question has to do with size of the human mouth vis-a-vis the mechanism at play in the harmonica, and the fact that in 200 years we have only found one way to really make this work. That is to say, to be able to get different notes out of an instrument by pushing air in opposite directions with your lungs, you need a an instrument with a system of individual, parallel reeds, that is small enough to hold in your hands and move across your mouth. The specifics here dictate that, in trying to invent a new instrument, you will basically always end up inventing the harmonica.