Let's start with the basics. The word "Magi" is indisputable. Everything about the language of all four Gospels indicates that they were written in Greek from the start, and the original Greek text in the second half of Matthew 2:1 says:

ἰδοὺ μάγοι ἀπὸ ἀνατολῶν παρεγένοντο εἰς Ἱεροσόλυμα

(In Latin letters) idou magoi apo anatolon paregenonto eis Hierosoluma

(In literal English) behold Magi from the direction of the dawn were arriving to Jerusalem

So there are your Magi from the east, and now we can get into the more complicated stuff.

The best question is not "Were these men truly magi?" but "What did Matthew mean by Magi?" We have no way to answer the former because we have no way to know whether or not the story is true. Even within the Gospel narrative, the story of the Magi is on shaky ground. Matthew is the only source for it. It doesn't even appear in other non-canonical early Christian books.

The author of Matthew seems to imply the Magi in the sense of followers of Zoroastrianism (as it was understood in the Roman world) through their use of astronomy/astrology. Ironically, this isn't actually accurate to actual Zoroastrian practice. There was no particular emphasis on astronomy among the actual Zoroastrians, but during the Hellenistic Period, the misconception became widespread.

The prophet Zoroaster himself was misunderstood as a Babylonian (hence the "Chaldean" ending on the Greek name Zoroastres), and conflated with Babylonian astronomy. He was actually credited by some authors with inventing the practice. As a result, his followers, correctly identified with the Magi were also associated with Babylonian astronomy. That's the sense that Matthew leans into.

Strabo, and Plutarch both provide descriptions that imply that the Roman world had finally come to understand the basics of what their eastern contemporaries believed after a few centuries of regular contact.

Of course, the Greeks and Romans hadn't been completely ignorant up to this point, but there were many misconceptions, which did continue to circulate and evolve into the medieval period. One of the earlier examples of a Greek writer understanding Zoroastrian beliefs is the Oracles of Hystaspes. This is an apocalyptic text, written at some point during the Hellenistic period, but attributed to Hystaspes, the first king to convert and shelter Zoroaster. In a way, it's very similar to a Zoroastrian book of Daniel.

The Oracles are mostly lost to us today, but seem to be a fairly accurate assessment of the end of days as described in Zoroastrian literature, called Frashokereti. We know it was discussed and heavily quoted by early Christian authors Clement of Alexandria, Justin Martyr, and Lactantius, who all actually accepted it's prophetical nature and interpreted it as describing the fall of Rome and the return of the Messiah. So by the second century we can already see examples of Christians identifying with Zoroastrian messianic beliefs. This may give us a hint to how they understood the Magi in Matthew. One valid interpretation is that the Magi came and acknowledge Jesus as the fulfilment of their own religious prophecy. This would also have been fuel for the pro-gentile camp when early Christians were still debating whether or not to proselytize non-Jews.

It's inclusion in Matthew is very interesting. On one hand, Matthew is widely considered to be the "most Jewish Gospel" because of how it accepts and promotes the use of Torah law, albeit under a heterodox interpretation. This has been interpreted as evidence that Matthew was writing for a largely Jewish-Christian audience, possibly even in Judea. On the other hand, Matthew also contains an early plank of early Christian anti-Judaism in 28:15, where it says (in Greek) that the "Ioudaiois" (Ἰουδαίοις) reject the resurrection of Jesus "to this day." The Greek word Ioudaoios can mean "Jew" writ-large or "Judean" as in a resident of the province of Judea.

If Matthew was writing for an audience that still considered itself Jewish, as most of the book suggests, then the traditional translation as "Jews" can potentially conflict with our understanding of the authors intentions. If it is meant as an explicitly geographic marker, then we just have to shift the author's location. It can be as small a shift as Jerusalem to Galilee or as large as going with the traditional belief that Matthew was written in Antioch. The idea of Antioch presents a tempting possibility, as the more cosmopolitan setting would have provided more opportunity to encounter actual Magi, or at least Zoraostrians.

Even though it doesn't factor directly into the conversation about Matthew, I think your last question is worth addressing too.

Was magi just an expression back then?

Yes. The Greek world absolutely did have some fundamental misunderstandings of the Magi, including their association with magic, as it is in fact the root word of "magic." By the Roman period, it was sometimes used euphemistically, but "Magi from the east who watch the stars by night" would still probably have been a clear enough reference to other established knowledge of the magi to make it clear these weren't random magic users. Matthew's description includes enough other details to emphasize their role as Zoroastrian priests to his audience.

However, another book of the Bible seems to use it in the "magical" sense. Acts 8:9-13 tells the story of Simon, a Samaritan who is described as "doing magic" with the the Greek word mageuon (μαγεύων). It's just the word magos transformed into a verb. Simon is actually a very prominent figure in other early Christian writing, and nothing about him or his following implies a Zoroastrian connection.

Acts 14:4-12 describes an encounter between Paul and a person described with these lines:

When they had gone through the whole island as far as Paphos, they met a certain magician, a Jewish false prophet, named Bar-Jesus... But the magician Elymas (for that is the translation of his name) opposed them and tried to turn the proconsul away from the faith. (NRSVue)

In this case "Bar-Jesus" (meaning "son of Jesus") is using "Jesus" as the standard Latin translation of the name Yeshua, not the literal Jesus of Nazareth. The word translated as "magician" is Greek magos. In fact, the Greek phrasing is probably better read as Elymas the Magus. However, beign described as "a Jewish false prophet," mostly roles out any Zoroastrian connection. "Elymas" has occasionally been associated with the Arabic word alim, meaning "wise," so some commentators have suggested that Elymas the Magus may actually be the product of a Greek writer trying to convey the idea of a name that means "wise" by using the word magos.

Traditions of the Magi: Zoroastrianism in Greek and Latin Literature by Alfred de Jong

A History of Zoroastrianism, vol. 3 by Mary Boyce and Frantz Grenet

"The Zoroastrian Doctrine of Salvation in the Roman World: A Study of the Oracles of Hystaspes" by John Hinnel

The New Oxford Annotated Bible (NRSV translation)

"The Magi as Wise Men: Re-examining a Basic Supposition," by Mark Allen Powell

The New Testament by Bart D. Ehrman

A Critical and Exegetical Commentary on the Gospel According to Saint Matthew by William David Davides and Dale C. Allison

So in critical bible studies, we go on from the assumption that the texts contained in the bible are not works of historiography, rather literary works that contain historical information. Scholars use different criteria to decide what is likely real history, and what is unlikely to be real history. This is not objective, and thus many scholars will draw different lines on what is historical. Most scholars agree that, Jesus was a real person, baptised by John the Baptist, who lived and preached in the region of Galilee and was executed by crucifixion around the age of 30.

The why do even the most skeptical people believe that the crucifixion happened? Multiple independent attestations and it would not make sense to invent the story. Death by crucifixion is shameful to a roman audience, and not at all what a Jewish audience at that time expect from the messiah. So since no one in their right mind would invent that, it is likely to have happened.

Is there independent attestation for the birth narrative? Well, not really, it is presumed that either Luke was based Matthew, Matthew based on luke or both on a now lost hypothesized source called Q (and Mark). Moreover, the specific story of the magoi is only told/referenced in matthew anyway.

Tre are good arguments against the historicity of that part of the bible story though.

• How much allegory/foreshadowing it contains Like Jesus laying in a manger (from which cattle eat, like how Jesus' flesh is supposed to be consumed during the last supper) like how is death is foreshadowed by the magi giving myrrh, which is used to embalm corpses etc.

• Who was there to record it. Mary, Joseph and their neighbours were (very) unlikely able to write. The magi are never mentioned again and it is incredibly unlikely we have their writings if they are historical people.

The birth narrative is likely a mythical story, containing none, or very little historical information. This does however not mean it is worthless as it is a lovely and amazing story, filled with allegory.

Source: Studies in the Textual Criticism of the New Testament, by Bart Ehrman

Edit: formatting and clarification

Magoi were prototypically a Zoroastrian priestly tribe, yes, from Old Persian magush, Avestan moyu-. And that's reflected by Matthew's statement that they came from the east.

But this wasn't niche knowledge: the Magoi had an extended reputation outside their homeland. Magos is the origin of the word magic, after all (Greek magia)! By the time of Matthew, magos had been used in Greek for centuries to refer to both Iranian Magoi and hierophants active in the Greek world, made to sound more exotic by using a foreign word. Euripides, writing in the late 400s BCE, has someone saying that a character disappears 'through sorcery, the art of the magoi, or the secret attack of the gods' (Orestes 1494). The Derveni treatise (also late 400s) refers to hierophants in a Greek mystery religion as magoi who make incantations and offerings of food to exorcise daimones. (Caveat: a 2014 article by Amir Ahmadi argues that the Derveni papyrus' account of the activities of magoi describes authentic Iranian rites.)

Fritz Graf, in a piece on the magoi in the Derveni treatise, collects a number of other Greek references to magoi and sums them up as follows:

these passages present the μάγος as an itinerant religious entrepreneur, concerned with Bacchic initations that had an eschatological component ..., divination ..., healing and purification ... and strange supernatural acts ... With the Derveni μάγοι, these specialists share the concern with the afterlife ... and, if we assume that the speaker is not very different from the magoi, divination ... and initiation into mystery cults ...

For some, they [the magoi] were indeed priests of another culture, either seen as authoritative as in Herodotus and Xenophon, or as somewhat uncanny, as in the historian Theopompus who tells of their power to resucitate the dead. To philosophers such as Aristotle, they representated an alien but acceptable philosophy ... To others again, they were simply weird and sexually ambiguous figures; to think that they were not really Greek helped to save one’s own identity.

The upshot is that the Iranian Magoi enjoyed an extended reputation beyond Iran. They're more specific than 'just an expression', but in Matthew 2 it does make sense to read magoi as infused with a dual meaning: as a reference to a well known Iranian priestly tribe, but also as a byword for exotic hierophants who performed religious and magical activities.