According to this source the first level of the Colosseum (the only part one could really fall from the stands into the games) is 34' high. At this height, it's very likely that a spectator would suffer severe trauma (landing on one's head would almost certainly mean death) or otherwise life - threatening injuries.

EDIT: formatting

Gladiatorial matches are often misunderstood: they weren't orgies of blood and violence, they were spectator sports featuring highly trained combatants. You wouldn't be any more a part of the match for stumbling into the arena than a modern spectator at a boxing match would be.

The exception are the midday "intermission" events, but even those were executions of criminals, not just random guys.

Didn't Caligula force a section of spectators to fight at one point?