It's been an issue when a member has recused himself. I was just reading about Muhammad Ali's Supreme Court case to determine whether he could have conscientious objector status or would go to prison for dodging the draft.

That was effectively heard by an 8-judge Court, as Thurgood Marshall had recused himself because he had been solicitor general earlier in the case. For a long time, the court looked like being deadlocked 4-4, in which case the lower court's judgement would have been affirmed without an opinion and Ali would have gone to prison.