I gather that this happened during the early Renaissance, but I don't understand why neumatic notation had to go. It seems like it was fully capable of supporting polyphony. If the works of Leonin and Perotin could be written in this notation system, it seems like there isn't any good reason why the works of Byrd, Morley and de Lassus couldn't be written this way. What am I missing?
...and for that matter, where did the modern notation system even come from?
Edit: Grammar hard.
Trying to write Byrd's, Morley's or Lassus' music in neumes makes no sense to me. Their music is already WAY too different from the music that was notated with neumes. We could still go for their intended usage: learn the music first, and then just use neumes to get your bearings. But if we were trying to do what 16th century musicians did, presenting NEW polyphonic music, it's not going to work. There are just too many details that would be lost by using neumes.
Western music notation started as a mnemonic device for an oral tradition. Neumes didn't give you much more than clues about the contour of a melody. Notation was supported because of bureaucratic purposes, to be able to fix the repertoire of the church at the time. You can do just fine with no specific pitch or rhythm if you just want to be reminded of melodies you already had to learn.
Polyphony was a very important influence in the development of notation. If you now have several different simultaneous melodies, and are expected to pick up new music from notation, well, you are going to need more information.
By the time of Leonin and Perotin people were not exclusively relying on the old (not very standardized) neumes. They were using rhythmic modes, a fixed set of patterns probably derived from poetry. That practice was codified in De Mensurabili Musica (c. 1250), traditionally attributed to Johannes de Garlandia. It was a limited system, but at least you had now more information on rhythm.
The next big step was in Ars cantus mensurabilis (c. 1280), by Franco of Cologne. He proposed a system in which the duration of the note was clearly indicated by its shape.
Next came Jehan des Murs and Marchetto of Padua, offering further rhythmic developments in the times of ars nova (first half of the 14th century, Philippe de Vitry is the most often mentioned composer from the period). Their work was concerned with binary and ternary divisions. This implicates more precision while notating rhythm, having fixed units to work with.
What about pitch? Neumes weren't specific in this regard, either. It's easier to survive with them for a simple tune in a very specific style, but things get complicated when people go for different things, particularly with polyphony. The Enchiriadis treatises (from around the 9th-10th century) propose alternatives to identify individual notes by using a staff (with varying numbers of lines) and some peculiar symbols. No more "kind of a little higher/lower," you can now write specific notes (from a previously defined set).
From the 12th and until about the 17th century we find problems with which notes are used. See, people in the time of early neumes had a system of notes inherited from the ancient Greeks, and there were a lot of changes after that. People used the term ficta to refer to alien notes that were being introduced to music. The problem of having chromatic notes is something that neumes just can't handle.
People went for a notation system that allowed them to be specific. Neumes covers a few different types of notation that were not very specific in the details they could present. Those systems were a starting point, but people went for more powerful tools because of the type music they started to practice. Once they had better tools, they composed more complex music and required even more powerful tools...
Sources: