I'm listening to the great courses audiobook "Alexander the great and the Macedonian empire" by professor Kenneth w. Hart and was surprised when he explained that not only did the Greeks consider Macedonians separate from them culturally, but that the Macedonians spoke a language that was unintelligible to the surrounding Greeks. This surprised me as I always understood ancient Macedonians as a subset of the Hellenistic peoples while culturally very distinct, however, if their languages were not mutually intelligible this would suggest to me this is not the case. My immediate reaction then was to assume that Macedonians may have been part of a subset of the Illyrian peoples but then it was also explained that Macedonian was not intelligible with any Illyrian dialects either (I think that's what he said at least if I remember). This would be rather astonishing as it would lead me to believe that the Macedonians may have made up their own entho-cultural-linguistic category of the indo-European macro-group that is no longer present today. If anyone has more information regarding this I would find it extremely interesting.
Illyrian and Ancient Macedonian are both part of a large group of IE languages that are united by nothing more than incredibly scant attestation, and along with Thracian make up an especially poorly-attested group of languages.
First, though, it's important to establish a bit about how languages are classified into families. Mutual intelligibility might be useful to help make some first guesses about how a group of unclassified languages are potentially related, but that's about the extent of its usefulness. The key tool linguists use to demonstrate language families is the comparative method. Linguists gather extensive lists of vocabulary items (kinship terms, body parts, numerals, names of plants and animals, environmental features, verbs for daily activities, etc.) and look for "regular correspondences" between single items on a list. As an example, there's a regular correspondence between English initial f and Latin initial p: English foot and Latin pes, pedis, full and plenus, flat and planus, etc. These regular correspondences are necessary evidence for claiming that two languages are related to each other. Mutual intelligibility doesn't care about regular correspondences and is influenced by a whole host of other factors.
So, what's the evidence about Macedonian, Greek, and Illyrian?
Ancient Macedonian is incredibly fragmentarily tested. Fortson's 2004 handbook of PIE and IE languages has a chapter on "fragmentary languages", and in the section on Macedonian there is a single paragraph about the language itself. The short version of Fortson's short version of what we know about Macedonian is that it is clearly closely related to Greek, though the exact relationship (whether it is a cousin to Greek, or just a dialect with some unusual features) is unclear. Furthermore, there's no good evidence supporting a close link between Macedonian and Illyrian.
Illyrian is similarly fragmentary, but the term is complicated by the fact that no one really knows what the term refers to. There's no inscriptions of Illyrian, so all we have to go by are names and a few glosses attested in Greek or Roman sources, and Fortson notes that it is now accepted that these words are gathered from linguistically diverse sources and shouldn't actually be grouped together. If we know next to nothing about Macedonian, we barely even know what Illyrian was. It's commonly described as the ancestor of Albanian especially among Albanian linguists, but the evidence for this boils down to: (1) Albanoi was the name of an ancient Illyrian tribe, (2) the dates and location work out, (3) we don't know enough about Illyrian or Illyrians to show otherwise, and (4) Albanian is unique enough within Indo-European that it can't be clearly placed with any other attested or extant IE language.