This is an excellent and difficult question. It's perhaps the central question of historical philology (that is, the study of texts, which includes editing), and not just for music, but indeed textual analysis of any sort.
The short of it is that there is no systematic answer, no airtight way to distinguish right from wrong that will always help you. Rather, each editor has to test a given text against their own sense of what the musical style is, since it is style that sets the limits of what is possible vs. not possible. But, ultimately, this is a process that relies hopelessly on circular reason, since our sense of what is possible within a style is built up by what we observe music in the style doing. So whenever we see something that is outside the bounds of what we think the style is capable of, we are always faced with two solutions.
1.) Our sense of the style is right, and the text is wrong. We are simply dealing in the case with a typo or a mistake that doesn't reflect what the author actually intended.
2.) Our sense of the style is wrong, and the text is right. This is a more radical decision that means that we have to adjust our sense of style to include what we formerly thought was not possible.
Again, there's no airtight way to make this decision. It's always a judgment call.
Often times, what editing means is projecting yourself into the mindset of the scribe who wrote the manuscript you are dealing with. In that sense, if you encounter a strange configuration of notes, one has to think "if I'm writing this manuscript, could I arrive at this configuration of notes by trying to write something normal but making a mistake? Or would it take something more effortful to produce this?" One could imagine doing things like misreading a clef and systematically writing the wrong notes for a while page, or forgetting to change the key signature, or similar such things.
Let me give you two examples from my own editorial work that shows the difference. The first I think us a relatively simple mistake, the second is something a little more purposeful.
##Example 1: A Legitimate Mistake
Our first example is the aria “Se cerca, se dice” written by Johann Friedrich Reichardt in 1791. Here’s what one passage looks like in the source I was using (the only source of the aria that has survived). When I punched that passage into a modern musical notation software, I hit “play” and could tell that something was very wrong. Here’s what it sounds like as written (I’ve muted the vocal part so we can focus on the instruments). Once the strings go away at m. 49 (13 seconds in the recording), there’s some really dissonant stuff that really does not sound like Classical Music. My brain instantly went “no way, there’s no possible way that’s the sound that Reichardt is going for.” My first thought was that I had screwed up, so I checked every note. Once I realized the mistake wasn’t one that I had made, I went about a harmonic analysis. I realized that the Bassoon made zero sense; it was simply never playing a note that made good harmony with any of the other parts (the bad notes are shaded in red in my transcription here), not to mention the fact that it went up to a high Db, which is essentially unfeasible for the instrument (unless you are Stravinsky). What I realized was that whoever wrote this had simply forgotten to put in a bass clef, and thus the whole passage was incorrectly notated a perfect 5th higher than it actually should have been. So in my final edition, I simply popped in a bass clef at the end of the first line of music (shaded in red again), and the passage sounded just fine. Note how in both versions, the pitches are notated on the exact same lines and spaces in the bassoon part, it’s just that by changing to a bass clef, we change what pitches are meant by those lines and spaces. So forgetting that bass clef is a really simple thing for a scribe to do, but it changes SO MUCH about the final result.
One thing that was tough about the passage is that it is dissonant even when corrected. But the way in which it was dissonant at first was just impossible given my understanding of eighteenth century musical style, while the kind of dissonance you hear in the “fixed” version makes sense according to how dissonances appear in the style. We can especially tell it’s “fixed” because of the moments of consonance that happen within the fixed version, like the Eb chord in m. 51 and the Bb chord in m. 53. Whereas the whole thing was complete nonsense in the original version.
##Example 2: A Legitimately Bizarre Passage
Our next example is a composition based on the exact same poetry, this time written by a guy named Antonio Traetta in 1769. Here is the passage in question as I notated it in my transcription. I had a similar experience to the Reichardt where I punched in a passage, hit play, and alarms started going off. But in this case, I realized that what happened was no mistake, it was just bad music.
How could I tell? Well, let’s break the passage down into its component parts. Here’s what it sounds like when you mute the voice and play everything else. Sounds completely normal. Now, here’s what it sounds like when you mute the clarinets and play all the other parts. Also sounds fine. But if we play all the parts at the same time? HOOOO boy, now that is some awful sounding stuff!! It’s especially obvious if you slow the speed by half and isolate the voice and clarinets, as I do here.
So we have a situation where the music is clearly not mis-notated. If it was, then if you muted either the clarinet or the voice, things would still sound bad. What’s happened instead is that Traetta has written two lines that simply don’t work when played together (I’ll give my explanation as to why in a separate comment). He’s composed bad music.
##Summary
Both of the preceding excerpts are outwardly similar. I transcribe something and things sound off, so I know something is wrong. That’s my personal judgment as an editor speaking, based on my own experience with this musical style. The difference between the two situations is that the first has an easy explanation and a fix, it makes sense how that one tiny little omission of a bass clef produces a cascade of bad musical effects that are not the intention of the composer, and thus by fixing that one minor issue, we bring the music back to what it is supposed to be. The second situation, on the other hand, has no easy fix. If we were to make this music sound decent, we would have to completely recompose it; erasing either the flute line or the vocal line and writing it from scratch. So there’s no plausible explanation for why this music sounds bad other than it was badly written by the composer himself. And since it is outside the scope of a music editor to improve the music, then we simply must reproduce the original, even if it is awful sounding. And the situation is no different if we are dealing with an unusual configuration of notes that is cool sounding. The process of judging and evaluating the weird sound would essentially be exactly the same: we would look for explanations as to why the effect exists and how it relates to more normal practices. The only difference would be that, if we found that the passage was not a mis-notated or mis-copied, we would evaluate it positively instead of negatively. Likewise, the same factors are in play no matter what musical style or time period we are working with. It all comes down to judgment based on familiarity with and understanding of a given style.
I hope these two case studies are useful in illustrating how the process of critical judgment works when editing music!
