There are two theories about where applying the hook to bad acts originated. Both involve the Miner's Bowery, a theater hall in New York that opened in 1878. Entry 15 cents, 25 cents, 50 cents, or 70 cents depending on which seats you wanted to get; beer was free-flowing, and as was typical for such theaters, audience participation was the norm. Jeering and cheering were expected.

One theory of the hook (via historians Laurie Jr. and Green) is that the founder, Harry C. Miner, introduced the hook; a more vivid story, via the historian Bernard Sobel, is that the son of H. C. Miner -- Tom Miner -- introduced the hook himself instead, during a performance of October of 1903.

This was during an "amateur" night, where a sequence of amateurs would perform for prize money. Five dollars and loose change thrown on the stage. Performers genuinely hoped not just for the humble prize money but a chance to be "discovered"; one such person, Eddie Cantor, went on to a successful career including in movies; you can watch a video of him here doing a vaudeville routine circa 1923 to get roughly the idea.

Audience members could demand longer performances from people they liked; the Scottish comedian Harry Lauder, intending a 20-minute performance as part of a sequence of acts, was met with such applause and demand from the audience ("it stormed for two minutes to an empty stage") that he ended up extending nearly an extra hour.

Of course, not everyone was a talent. Returning to 1903, a particularly unfortunate nameless tenor was starting to get continuously booed by the rowdy audience. Mr. Miner had his property manager, Charles Guthinger, attach a long pole to a cane discarded from the previous act and allegedly had the performer dragged away by the neck (not actually, more on that in a moment). This was popular enough that at the next (also painful) performance, someone shouted "get the hook" and it became a regular device.

It was probably not by the neck; the cane that they would have had at hand would not have fit around someone's neck, but more likely went under the collar instead. The "shepherd's hook" style hook large enough for necks didn't get introduced until later.

Cantor, who performed under threat of hook, claimed he developed his movement style (which you can see in the video) as a way to avoid getting snagged; this was likely somewhat tongue in cheek, as it was easy to tell from audience reaction whether or not one was at risk.

This seemed to be mainly a Bowery-specific schtick; where cartoons enter the picture is that they were originally presented in the same manner as vaudeville, with fourth-wall breaking and loud audiences, and they felt license to use the same devices. I wrote more extensively about this here, in reference to The Case of the Stuttering Pig.

...

Hackman, F., McNeilly, D., Cullen, F. (2007). Vaudeville, Old & New: An Encyclopedia of Variety Performers in America. Routledge.

Monod, D. (2020). Vaudeville and the Making of Modern Entertainment, 1890–1925. United States: University of North Carolina Press.

Wade, S. C. (1898). A Birds-eye View of Greater New York and Its Most Magnificent Store: Being a Concise and Comprehensive Visitors' Guide to Greater New York. United States: Siegel-Cooper Company.

Weinstein, D. (2017). The Eddie Cantor Story: A Jewish Life in Performance and Politics. Brandeis University Press.