The parliment threatened, if Edward VIII didn't step down from the position of king (or simply not marry Simpson), to resign "en mass". What exactly would that entail? Would every member that disagreed with the marriage simply resign? Would the entire government be dismantled? Would the power go back to the King? Would it just be anarchy? I don't really understand the whole threat of the resignation, especially how important the threat truly was.
I think you've misread your sources. It was the government that threatened to resign from being in government, not the entirety of the parliament.
The government of the UK consists of the Prime Minister as chosen by parliament from its midst, and the cabinet ministers appointed by her. They would have remained members of parliament.