When 2/3rds of the cast suddenly drop dead, was this a motif people would go into the theatre anticipating or was it designed to catch them off guard?
Everybody-dies endings had some precedent in Early Modern drama before Hamlet. One literary tradition to which Hamlet belongs is the Senecan tragedy, a Latin-inflected adaptation of Greek tragedy with Stoic themes, elements of revenge and the supernatural, and onstage bloodshed. Early Modern drama cribbed its structures and themes from Seneca the Younger's plays (think Phaedra, Medea, Oedipus...) and these works would make up the Classical pattern for tragic drama that would be familiar to university-educated playwrights, moreso than the Greek-language originals. Titus Andronicus predated Hamlet in Shakespeare's own revenge-tragedy oeuvre and it bears the most direct influence of Seneca's works in its presentation of a cannibal banquet cribbed from Thyestes. Seneca's plays weren't the only influence on English dramatists -- Shakespeare and his contemporaries cribbed from English chronicles and Italian writers for both the premises of tragedy and the unique forms of violent death depicted on-stage. English history plays like Marlowe's Edward II incorporated famous nasty deaths, plays with Classical themes like Shakespeare's own Julius Caesar and Coriolanus killed off their title characters, but the Senecan revenge tragedy brought together grave philosophical themes and grisly subject matter in the tastiest possible combination.
In the 1580s, Thomas Kyd's Spanish Tragedy brought the Senecan tragedy back to the English stage and set a precedent for dramas concerned with revenge, incorporating supernatural themes alongside high-killcount endings. This play alone would have made the ending of Hamlet not such an unprecedented marvel, and its themes were still in circulation in a more refined form decades later, suggesting they remained popular or at least in the tragedian's dramatic repertoire without growing completely stale. Revenge tragedy was having a real cultural moment in the end of the 16th and start of the 17th century, perhaps as a manifestation of political anxieties and domestic turmoil -- both companies of adult players and companies of boy actors tackled the genre with plays like Marston's Antonio's Revenge, Marlowe's Jew of Malta, and Middleton's borderline-parodic Revenger's Tragedy. It's a little anachronistic to imagine Elizabethan/Jacobean audiences approaching these plays as their own discrete genre -- a little like superimposing the category of "film noir" on movies that 1940s moviegoers would have identified only as B-pictures, crime films, or melodramas -- but by the end of the 16th century the structure and elements of revenge drama were established enough to be the subject of parody. The 1599 tragedy A Warning For Fair Women roasts the established pattern for revenge tragedy in its introduction, where the personification of Comedy critiques its opposite as having nothing to offer but:
How some damned tyrant to obtain a crown
Stabs, hangs, empoisons, smothers, cutteth throats,
And then a Chorus too comes howling in,
And tells us of the worrying of a cat,
Then of a filthy whining ghost,
Lapped in some foul sheet, or a leather pilch,
Comes screaming like a pig half stickèd,
And cries Vindicta, revenge, revenge
(Which is all well and good for A Warning For Fair Women, which is not a revenge tragedy but a domestic tragedy -- it has no highfalutin tyrant at the center of its action but a real-life murderous love triangle.) The sense that tragedies ended one way and comedies ended another way was already well-established among theatergoers, even if the parameters for a happy ending looked a little different than in modern theater. The tragedy/comedy/history categories Shakespeare's plays find themselves slotted into are a later addition, in some ways as bogus as attempts to shoehorn any complex work of fiction into a single genre category, but there was still recognition that certain stories ended well for their characters and others badly -- if Hamlet ended with Fortinbras ripping off a fake beard and marrying Hamlet in a double marriage alongside Horatio and Ophelia before pardoning Claudius and Gertrude, or if As You Like It ended with a multiple homicide and an offscreen burning at the stake, that incongruous ending would be displeasing rather than cathartic.
It would have been difficult to mistake Hamlet for a pastoral romance or a sexy city comedy even from its first scenes, taking place on a stage set up to announce that day's show would be a heavy one, and a tragedy about murder and usurpation was effectively obliged to end with some kind of bloodshed. Hamlet wouldn't have been old-hat by the time it premiered -- it's more than just a rehash of The Spanish Tragedy, and it's perfectly possible to enjoy the play if you're spoiled for the ending -- but the ending wouldn't have come as a complete blindside in a play clearly earmarked as a tragedy. The audience would have anticipated some kind of violent conclusion from at least Act 1 Scene 5 onward, but while audiences might anticipate that somebody would die at the end of the play, they might not have anticipated exactly who or how -- they might have a reasonable expectation that Claudius would receive his just deserts and that Hamlet might die avenging his father, but the specific manner of death for Hamlet, Ophelia, Gertrude, R&G, Laertes, etc. would have still carried some pleasing novelty.