Your question is unanswerable because its central premise is flawed.

With Genesis, the change is easily explained - Peter Gabriel left as bandleader, Phil Collins took over. Phil Collins has a different style and sensibility, so naturally their work reflected that.

Rush released their first record in 1974, so to say that they "sold out" around 1975 is pretty ridiculous; what else do you have to compare it to? And in what universe is that material more radio friendly? As the 70's wore on, their songs became longer and longer and incorporated more and more synthesized elements and odd time signatures.

For Camel, you get an orchestral concept album in 1975, which again goes against your conceit. And their material became more jazz-influenced, which in the late 70s isn't exactly a big seller. You might as well say that Steely Dan was getting more commercial with Aja, or that Joni Mitchell was getting more commercial with The Hissing of Summer Lawns.

If you could provide more examples we could possibly answer your question, but as it stands your question is based on a flawed premise. As bands go on, they tend to get better at what they do and often have access to better technology due to bigger budgets and improved recording technology; oftentimes this can give the illusion of "more commercial" when compared to earlier work, when it really just means "played better, recorded better, in better facilities". Most of the time, though, "My favorite band sold out!" translates to "I liked the way they sounded before."