Situations in which you could decline depend somewhat on the period, but there are a few general themes you can find.

• If challenged by a woman you could decline without any worries.

• If you were infirm and elderly, you generally could decline, but there was also an implicit understanding that one of your more spry male relatives, a son or nephew for example, would take your place in defense of family honor.

• If you were well known for being religiously devout, you generally got a free pass in declining (the churches taking a very dim view of the practice), although there also, of course, was something of an expectation that you wouldn't be living your life in the kind of way to incur such challenges in the first place.

• If you were royalty. The King would not be expected to fight a duel. Nor, really, should the Prince of Wales, although it should be noted that his brother, the Duke of York, actually did duel in 1789, having made a declaration that he would not hide behind royal privilege over his dispute with Col. Lennox. The Colonel while deemed technically proper in his conduct, nevertheless recieved censure from his unit for a lack of judgement, and the Prince of Wales snubbed him at a ball held soon after. Had he not missed, he may very well have proven the exception to the rule that illegal duels were mostly unpunished, but we can only speculate.

• Similarly, you could not be held accountable for something done in your official capacity, and were within your rights to decline, but that right would often be waived. We see this with military officers, famously the cause of the duel fictionalized by Conrad and later Ridley Scott, and politicians, best exemplified by Sec. State Clay's challenge of Sen. John Randolph over remarks the latter made on the Senate floor resulting in a challenge and Randolph waiving his privilege to decline, which was considered essentially a Constitutional Right under Article I, Section 6(1) "and for any Speech or Debate in either House, they shall not be questioned in any other Place."

But now we get to the main event, as the biggest one is social disparity, as equality of the duelists was both the most important element of the duel, and at times the hard to be sure of, especially in the United States, but even in a place like England where impoverished farmers in the country might nevertheless be nominal members of the landed gentry. In short, you could decline to duel someone who was your social inferior - in some situations if they were impertinent to you the expected response was simply to thrash them with your cane (thrashing has a lot of social cached because of that, see the caning of Sumner, for instance) - but it wasn't always clear if someone was since the 'dueling set' was binary, and thus included within it a number of subdivisions which could represent vast disparity. So it could be tough, sometimes, to determine just whether someone was a person who you could duel. In the 1838 code, a specific provision was included for figuring this out if you didn't know who the person was:

If a stranger wishes you to bear a note for him, be well satisfied before you do so, that he is on an equality with you; and in presenting the note state to the party the relationship you stand towards him, and what you know and believe about him; for strangers are entitled to redress for wrongs, as well as others, and the rules of honor and hospitality should protect him.

One famous duel to occur came about because Rep. Jonathan Cilley declined to duel the publisher James Webb, both because he thought the man disreputable, as well as because as a member of congress, be felt covered by the aforementioned privilege. Rep. William Graves, a fellow member of Congress, was offended by this refusal since he had delivered the challenge, and felt that implicit in the refusal, Cilley was saying Graves wasn't a gentleman. So Graves in turn challenged over that offense, and Cilley was cut down in the exchange despite the disagreement not even being between the two men.

So anyways, the point is that you could decline based on social disparity, but you needed to be sure of it, and also confident that others would agree with you, which certainly resulted in either situations like Cilley's, or else situations simply where the challenge was accepted to be better safe than sorry, although of course that ran the risk that everyone would have agreed, and now you're dueling a disreputable person which is actually bad for your reputation! A big part of the role played by your Second, then, would be in helping you to make that decision to avoid the wrong choice.