I think you're starting from a position of wrong assumptions. We have music notation from the travelling troubadors as well as music notation of the "free form choral music" you're referring to, which I assume you mean plainchant or Gregorian chant. There's even Ancient Greek music notation which we don't know how to read, and Byzantine music notation.

You seem to think that chanting is improvisational, and it's not. Plainchant has over a thousand years of history, and we have tons of written examples. Music notation exists in many cultures, and dates back thousands of years. It didn't just spring up out of the ground.

It seems like the history isn't all figured out

Yep. It's a puzzle of something we don't really know, and many pieces are missing.

I don't understand the transition from traveling singers and long free form choral music to having compositions written out.

Those things are not mutually exclusive. We have plenty of music that was notated but still required "free form" things.

Writing is a form of technology. So, let's think about what notation is good for, and why people (in the middle ages) would start using it instead of just free forming everything.

Notation started as a tool for people to remember some music they already knew. Why? Because people wanted to put some order to the music that was sung in religious services, as in "we must sing the right music, in the right way." Good luck free forming stuff and wanting for people to sing the right music in the right way all over a continent. People were trying to give some persistence to music, not an easy task.

Also, people were (at some point) into singing simultaneous melodies. That adds another problem, because you now have to coordinate two things that happen at the same time. You can free form that, sure, but if you start doing fancy things you need to sing with some other people who are capable of doing those fancy things.

If you come up with something nice and want other people to sing it, well, it's harder to communicate things that have to happen at the same time. People started having more simultaneous melodies, and then those had different rhythmic patterns. It's difficult to work all those out in your head, it helps to have a tool to work things out without having people singing for hours until you find what you want.

How and why did sheet music/rhythmic notation become the norm

Has it become the norm? I think it has not. It is the norm for some musical traditions, but that doesn't mean all music has become exclusively notated.