So I would stress that while perhaps respectable, dueling was never legal (minus some very, very narrow, pedantic and unimportant exceptions). That said though, the illegality of dueling was not generally acceptable as an excuse to decline a challenge. As for the actually issue of declining, I've written a few things previously which touch on the topic.

I would point to this old answer and this one and this one all touch on the topic to some degree or other.