I was trying to find the answer to this last month, and this rather unsatisfactory article was the best I could do. But I tried to dig a little deeper, and this is what I came up with:

While audiences seemed to have vocalized their disapproval in various ways through history, the specific "boo" sound seems to have originated in Victorian England, likely in the context of melodramatic theatre. Audiences were expected to cheer and clap when a heroic figure took the stage, and boo and hiss at cads and villains.

The first real example of this usage, according to the Oxford English Dictionary, comes from the Christian World magazine in 1884, and was clearly related to melodrama: "The sibilant hiss and the contemptuous 'boo'." (The fact that this term is in quotes indicates that it was something of a novelty at that time.)

The sound seems to have been associated with that of lowing cattle - in English, we now say "moo", but also have said "boo" (the Dutch, for example, still say boe).

It seems to have spread to other parts of the English-speaking world by the early 20th century - here's an example from New Zealand in 1912.