Specifically, why did Whorf and Knorozov think its syllabic? Even knowing it is syllabic

Knorozov counted the combinations of Mayan letters (unigrams, duograms, trigrams, quatrograms, pentagrams) and presumed them to be "words" with the 355 letters corresponding to the amount of syllables found in the Mayan language.

He concluded that the alphabet transmitted by Diego de Landa spelled out the Spanish names of the letters and not the letters themselves (think "el-le" for "l") and he substantiated that by the identification of Mayan Gods.

/u/Freiheit_Fahrenheit covered the basics of it. Essentially, prior scholars assumed that Maya hieroglyphics were entirely logographic. The significant step in deciphering them was realizing that certain elements were repeated - specifically elements on the left, top, right, and bottom of the glyphs. It turned out that the number of unique elements corresponded (more or less) to the number of unique syllables in the Maya language. The syllabic nature of the symbols helped explain the "alphabet" provided by Diego de Landa. Between this and comparisons to modern Maya dialects, scholars were able to tease out part of the language from the glyphs. And from there it was simply a matter of time and effort by trained epigraphers and linguists before most of the language was cracked. Although not all of the glyphs have been translated, and other scripts in Mesoamerica haven't even remotely been cracked yet. (Zapotec is only partially deciphered and Epi-Olmec is a complete mystery.)

I understand picking up an entire book on the subject is a bit much for simply a passing curiosity, but if you want something more detailed, I'd recommend the 1-hour Nova documentary Cracking the Maya Code, which is available on youtube for free.

One important clue came from Fray Diego de Landa's Maya "alphabet," as /u/snickeringshadow and /u/Freiheit_Fahrenheit pointed out. This was included in a document de Landa wrote called Relacion de las Cosas de Yucatan, which provided a lot of ethnographic information about indigenous Yucatec Maya culture but was largely written to defend de Landa against charges that he'd been too hard on indigenous converts by carrying out an Inquisition against them without proper authority. Here's how the "alphabet" happened. Basically, de Landa sat down with a Maya informant, a nobleman named Gaspar Antonio Chi, and asked him how to write the names of the letters in the Latin alphabet as they were pronounced in sixteenth-century Spanish. That's as opposed to the sounds those letters represented. For vowels, the names are the same as the sounds they stand for. The letter a is pronounced "ah" and, in Spanish, is always pronounced like "ah" (as opposed to in English, where a can stand for different sounds depending on the word). The letter name for b is "be," which sounds like "bay" to an English speaker. Be is the word for "road" in Yucatec Maya, so de Landa's informant wrote down the logogram (word-sign) for be. To take another example, the Spanish name for h is "hache," which is pronounced "ah-chay." De Landa didn't ask Chi how to write the breathy h sound; he asked him how to write "hache." What Chi wrote down for him was the second syllable of the Spanish letter-name, che. And besides cases like that, mixed in with the other "letters" in the "alphabet" are signs for the consonant-vowel syllables ka, ku, and k'a.

Even more tellingly, de Landa provided an example of a sentence written in hieroglyphs. You get the impression that de Landa said, "Hey, write down a sentence in Yucatec" and that Chi was about fed up with him, since the sentence is "I don't want to": *Ma' in k'ati," literally "It is not my wish [to do so]". As written, the sentence includes signs for syllables (ma, k'a, and ti) that aren't included in de Landa's "alphabet." All of this added up to a pretty suggestive case that syllabic signs had played a major role in the Maya script.

So Yuri Knorosov looked at a medieval Maya document, the Dresden Codex, and noticed that there was a picture of a turkey in it. The Yucatec word for "turkey" is kutz, and the picture was labeled with de Landa's sign for ku plus a second, unknown glyph: ku-X. That same unknown glyph was the first glyph in a caption labeling a picture of a dog, and the word for "dog" in Yucatec" is tzul. If glyph X was the syllabic sign for tzu, Knorosov suggested, one could read the label for the dog as tzu-lu, or tzul. And there was another context where the number eleven (buluk) was written with another unknown glyph, plus lu and ku (Y-lu-ku). Knorosov proposed that glyph Y was the syllable bu. He went on in that vein, working out suggested values for unknown syllabic signs based on context and the ways they were combined with known signs.

The full story is way more complicated than that, of course. I would recommend reading Michael Coe's Breaking the Maya Code for a fuller account, but that's the general outline of how the syllabic part of the script was identified. I should point out that most Maya characters are logograms standing for whole words, not syllabic signs.

EDIT: k'ati, not k'a ti.