It's important to remember that movies will not be as quick to show the evasions, deflections, accommodations, and apologies that prevent most challenges from reaching the point where anyone actually engaged in combat. You're right that social norms of the 17th-19th centuries could give men few options other than issuing or receiving a challenge. However, media less commonly portrays the important work done by seconds. By acting as intermediaries between the two duelists, seconds could negotiate a cessation of hostilities. Perhaps the challenger was drunk, or could plausibly have been drunk when he made his challenge, and on that grounds could retract his statement. Or maybe the challenged could point out that dueling was illegal in the place he was visiting, or that he was under military orders not to duel. With the heat of the moment passed, even a private, written, or public apology could be worked out.

In short, it came down the personalities of the duelists, the nature of the offence, and the social pressures both found themselves under to carry the dispute to a violent conclusion. Cohen, By the Sword has some good chapters on dueling, including some takes on the percentages of of men who dueled and the mortality rates among duelists.