So from what I understand, in early medieval Norse culture if you were challenged to a holmgang, you had to show up no matter what or the challenger was proven correct in their insult towards you.
But what if the challenger was a douchebag who just wanted an excuse to kill people and everyone in the community knew it? What if you were old or sick? Was there any way you could tell the challenger to fuck off and for you to keep your pride?
The whole system just seems really antithetical to making a working society and I figure there must be some exceptions.
The concern you point out is very real in the Icelandic Sagas. Some characters, at least within the sagas, do go around challenging people to duels in order to gain material wealth. Some legal cases could be settled by combat and unscrupulous characters could try to force combat against other parties in legal cases. One such character is Dueller Bersi, this epithet is derisive or at least pejorative. And basically the only thing stopping someone from running around starting frivolous lawsuits and then killing the defendants in a duel was reputation. Even Gunnar, one of the most magnanimous and admirable saga heroes, is castigated for challenging an older man to a fight over a legal dispute in Njal's saga. Although Gunnar feels he is in the right, his reputation is damaged by the affair. The entire justice system lacks any sort of executive function in saga age Iceland. So justice depends on fighting ability and the reputation required to ensure the aid of other good fighters. Dueller Bersi wins wealth with his behavior but ultimately isolates himself from his community. He then becomes vulnerable to group retaliation. Dueller Bersi could be the greatest fighter around, but if I'm very likeable and have a good enough reputation there's a good chance I can get forty men together and burn his house down in the middle of the night with him in it with little fear of repercussion. No one will miss him, there are no police to investigate. Someone would have to like Bersi enough or at least feel honor bound to do something about it. But Bersi is a dishonorable man or at least infamous so almost no one will feel compelled to make an issue of it. Conversely if I don't have any friends I'm out of luck. And this also means that I have to guard my honor or reputation very closely or risking losing access to the protection of this kind of retributive vengeance/justice. In Njal's saga for example we see that entire feuds are started over what might seem to us as trivial insults. Because to accept these insults is lose reputation and become vulnerable. I think this general idea can be seen at work in Njal's Saga and Egil's Saga.