The common narrative is that the code was never cracked, but how much did Japanese military intelligence figure out? Did they figure out that it was based on an obscure Native American language that they had no access to knowledge on? Did they partially crack it mathematically? Or were they completely in the dark? How effective was Imperial Japanese counterintelligence in general?
They never came anywhere close to understanding or cracking the code.
It is thought they seemed to understand or at least have a suspicion it was Navajo because in 1943 they'd captured a Navajo American soldier called Joe Kieyoomia in the Phillipines. First they beat him and interrogated him because they thought he was Japanese-American (this was fairly common - some were even almost shot by their fellow Americans). But eventually they took his word that he was in fact a Navajo American, and two women visited him in the POW camp and presented him with a list of transcriptions they had tried to write down from what they'd heard on the radio. Most of it meant nothing to Kieyoomia as it's almost impossible to write down some of the sounds so it was a terrible transcription, plus the few words he did recognise and translate meant nothing to the Japanese as they were just animals like 'snake' or 'deer' or whatever. They tortured him but seeing as he didn't know the code, there was nothing he could do to help them.
The important point about the Navajo code is that the Code Talkers couldn't only speak Navajo - it has to be a code based on it as they knew a Navajo soldier might well get captured, serving as they did all over the Pacific. So the code was 17 pages long and used Navajo words for animals that every Code Talker understood to match with an English letter, so for instance the Navajo word for "cat" (moasi) would be 'C'. Some words also referred directly to phrases like 'ship' or 'submarine', and they used shorthand, meaning some letters would be ocassionally omitted to remove consistency from the code. To add to the difficulty, it's a tonal language (akin to Mandarin) and has complex declination of cases (like Finnish or Hungarian), and it's an non-written language.
The only possible way the Japanese cryptographers would have had any chance of decoding it is if they had captured a Code Talker.
There's a great book on the history if cryptography by Simon Singh called The Code Book that mentions them.
Follow-up question: has the memory of Dine code-talkers affected efforts to continue Dine as a community language?