From scene I.ii:
BOTTOM: Let me play the lion too: I will roar, that I will do any man's heart good to hear me; I will roar, that I will make the duke say 'Let him roar again, let him roar again.'
QUINCE: An you should do it too terribly, you would fright the duchess and the ladies, that they would shriek; and that were enough to hang us all.
ALL: That would hang us, every mother's son.
Of course, these characters were intentionally written to be very silly, and Shakespeare himself was never hanged for the many frightful scenes in his plays, so it can't be an entirely justified fear.
But the play-within-a-play scenes are funny because they make fun of actual phenomena—Bottom parodies the conceited leading man, Flute parodies the teenage boy embarrassed about playing a woman, Quince parodies the frazzled director, and so on. What was the real-life basis for the joke where the actors are afraid of being hanged for being too frightening?
Well, there are a few ways of tackling this question: primarily, what would it mean in Shakespeare's time, and what would it mean in ancient Athens, where the Midsummer takes place (the in-universe approach)? I'm gonna focus on an in-universe explanation, because that's just how I be. Within that framework, though, there are a couple further approaches: what would Theseus do, and what would happen according to the procedures of Ancient Greece?
There are a couple reasons for this distinction. The first is pretty obvious: Theseus (presumably) wasn't a real person. He was a hero of Greek mythology who is credited for founding Athens and uniting it with the rest of Attica, amongst many other famous labors. Shakespeare draws from that mythology to build a world in his play set in a vaguely-defined ancient Athens, where Theseus is getting married to Hippolyta the Amazon and seeks wedding entertainment (there's also a minor subplot of some fairies and mixed up lovers, but who cares about that plotline?). The second reason is that, even if Theseus lived, this would have been centuries before theatre emerged in Greece. Theatre didn't become a thing in Greece until closer the late sixth/early fifth century Athens, whereas if Theseus did exist, it would have been in the Late Bronze Age, before the 8th century.
Unlike many of Shakespeare's plays (believe it or not), Midsummer is pretty original: it isn't an adaptation of an historical event, or poem, or myth, or other famous story. It does, however, draw inspiration and incorporate aspects from a variety of sources, such as a story from The Canterbury Tales and Plutarch's biography of Theseus. The story of Pyramus and Thisbe that the Rude Mechanicals perform, meanwhile, comes from the Roman poet Ovid's Metamorphoses, written in 8CE. (Metamorphoses is supposed to be a history of the universe, and in fact places the events of Theseus after Pyramus & Thisbe.) If I'm not mistaken, I think actors in Shakespeare's times dressed more contemporaneously (i.e., in Elizabethan garb) rather than in period clothing (e.g., like ancient Athenians), so while Shakespeare's story was inspired by the characters of Greece, he may have played a bit more loosely in defining the rules of that world, instead opting for something that his audiences would understand and relate to better. [If any Shakespeare scholars can comment on how accurate this claim is, I'd love that.]
If we said in-universe took place while theatre was thriving, then I think their fear was exaggerated, but rooted in something real. During the fifth century and some of the fourth century, theatre in Athens was primarily at an annual festival called the Dionysia. Theatre also occurred in smaller festivals elsewhen across Athens, as well as outside of it, but it was biggest at Dionysia. It was a competition where three playwrights submit a set of four plays that are performed, and at the end of the festival judges vote on the winners. (At risk of getting too in the weeds in literary analysis, I don't think the Rude Mechanicals would even make it to Dionysia, since they… aren't very good. But we can ignore that.) While travelling performers of sorts did exist in Greece at the time, there wasn't a whole lot of people gathering together to perform a play that the king will witness. In the Hellenistic period (321-31 BC) we saw a lot more of theatre troupes banding together, traveling across the land, and making performances for special occasions, rather than the typical competition plays.
But when we look at the content of Quince's Pyramus & Thisbe, only two interpretations are plausible if we place it in context: either he never would have written it in the first place, or he probably would have scarred people (or at the very least, left them displeased by the story). I don't know if the lion in particular would have been a problem, but later in the play Pyramus and Thisbe stab themselves (poorly) to death, on stage. And this was a big no-no in Ancient Greece: playwrights did not include violence or other graphic moments on stage. While common elements of the story, this always happens offstage, to be reported to the characters through a messenger or otherwise explained. This is why in Sophocles's Oedipus Rex, for example, Oedipus and Jocasta both run offstage and Oedipus pokes his eyes out while Jocasta hangs herself. Oedipus later returns blind, explaining what he's done, while someone offstage finds Jocasta's body and reports her death to the people onstage. The Greeks did not think seeing gore and violence and other such gruesome stuff was proper in theatre; their successors, the Romans, of course had a much different perspective. So either Quince would know this rule and not have written the play as is, or he would try to be "innovative" (which is kinda believable, actually) and decided to include those scenes, only to upset the audience.
I'm not sure if they'd actually be punished for doing this. But in Classical or Hellenistic Athens, it would likely not be well-received, and at the very least he probably would have lost at whatever festival he was competing in. Though, again, the Rude Mechanicals put on a bad performance, so that was likely happening regardless of if they follow the rules.
But if we lean into the anachronism and say that this is King Theseus's court, what would happen? That's a good question, and I can only speculate. The only relevant thing I can glean from Plutarch is that Theseus was disturbed by the suicide of his friends, so I can imagine him not being a fan of seeing that on stage. What kind of reaction he would have or response he would dole out, though, is beyond me.
This was an interesting question for me: Midsummer is my favorite Shakespeare play, and I'm fascinated by Ancient Greek theatre, yet I've never thought about analyzing them together (which, now that it's come up, seems like the obvious thing to do). In writing this, it's just now occurring to me how anachronistic the whole play is, which makes it difficult to peg a specific answer to the question. I think a lot of it boils down to what I said earlier about theatre story vs rules: Shakespeare made a story set in Athens, but the "Athens" he created was more a reflection of the Elizabethan world that Shakespeare and his audience lived in, rather than a historically accurate presentation of Greece. If I were more familiar with how Shakespeare's plays were written and staged, I think that would make for a fascinating cross-analysis of the two interpretations.
Further Reading
Green, J. R. (1996). Theatre in Ancient Greek society. London: Routledge.
Kotlinska-Toma, Agnieszka. Hellenistic Tragedy : Texts, Translations and a Critical Survey, Bloomsbury Publishing Plc, 2016.
Muir, Kenneth. "A Midsummer Night's Dream." The Sources of Shakespeare's Plays, Taylor & Francis Group, 2005.
Plutarch. "Life of Theseus." Parallel Lives. Trans. Bernadotte Perrin. Cambridge: Loeb Classical Library, 1914. https://penelope.uchicago.edu/Thayer/E/Roman/Texts/Plutarch/Lives/Theseus*.html