There have been many analyses of this, but I haven't seen anything about this from the perspective of the Argentinian junta, so it's hard to know what was really going on.

In general, deterrence threats (e.g., "you do X, I will do Y, so don't do X") are only going to work if they are credible. "Credibility" is tricky: it's usually thought of in terms of technical capability (e.g., I could nuke you), but it's also political (will you nuke them?).

In the case of Argentina, there are two options. One is that they didn't think the UK would nuke them, which would have been a fair assumption (the public outcry would have been huge; it would have been way overkill for the UK; etc. etc.), and ultimately was true. This is what I suspect what going through the minds of the Argentinians, if they thought about the nuclear question at all.

The other possibility is that they thought the UK would nuke them and did it anyway. Is this totally absurd? Maybe. But there have been examples of this happening in the past; Saddam Hussein was pretty convinced that the US might nuke him because of his actions in Kuwait (burning oil fields, etc.), but did them anyway. This messes up deterrence theories because they usually assume "rational actors," but real-world people are often not super rational, especially if they are dictators and their irrationality can actually be made into rule of law. I just bring this up as an alternative scheme.

Where does the Argentinian junta fall? Somewhere in between, from what I have read. They believed they had a chance at taking the Falklands successfully, despite going up against an impressive military power. They knew there might be consequences but overlooked them because of their pride. They didn't seem to think they'd get nuked, which was correct. They did seem to think they'd be successful, which was wrong.

In general, deterrence theorizing has changed a lot over the years, but basically nobody has believed for a long while that nuclear weapons deter conventional attacks (in Korea, China attacked the US; in Vietnam, the North Vietnamese attacked the US; Iraq attacked both Israel and the US in the 1990s). Having nukes does not grant you an "instant win" in a conventional war (see also: Korea, Vietnam, Afghanistan, Iraq), and since nuclear weapons got established as "special" weapons (the history of the "nuclear taboo" is complicated; it wasn't instantly the case), the barrier to using them has gotten ever higher.

MAD is a situation that only applies for two nations that hold themselves in mutual vulnerability — something not the case in nuclear vs. non-nuclear power wars.

On the "nuclear taboo," see esp. Nina Tannenwald, The Nuclear Taboo. David C. Gompert, Hans Binnendijk and Bonny Lin's Blinders, Blunders, and Wars Book (from the RAND Corporation) has a chapter on the junta's thinking in the Falklands/Malvinas incident — it essentially characterizes them as "obtuse" as opposed to "irrational." It doesn't discuss nuclear matters at all, and doesn't seem like it has an "inside" story (which is what we'd really need to know what motivated them, scared them, didn't scare them, etc.).