There are two reasons -- one is that it's just been a very long time. The area that's now "The Arab World" was under the control of Arab-Islamic forces by the year 700 CE or so, so you're seeing the end product of a cultural assimilation that's been going on for well over a thousand years (compare to how quickly the Americas became Anglo/Spanish/Lusophone, for example).

The second is the cultural power of the Arabs as the bringers of Islam, with the sacred text in Arabic (the rationale for which is baked into the text--that God delivered a message to the Arab peoples, in their own tongue), so Arabic thus takes on the prestige of being a sacred language.

Being an "Arab" is primarily defined linguistically -- an Arab is someone whose first language is Arabic (depending on the area there might be some alleged descendance from one of the original conquering tribes, but the further away you get from the peninsula the harder to prove and more dubious such claims are).

To take but one example: Coptic, the linguistic descendant of ancient Egyptian, was still spoken in some rural villages in Egypt as late as the 17th century, for example, so Arabization was a very slow process. Also remember that literacy was very low in most of the region as late as the 19th century, so when we talk about languages being spoken, we mean that literally -- people could speak Coptic and Arabic, but only a certain small subset of the population could write either -- and, unless they were going into the Coptic Christian clergy, the useful language to learn was Arabic because it was the language of bureaucracy. Most Arabic speakers couldn't speak or read Coptic, but nearly all Coptic speakers could understand Arabic--so when time comes to develop a national educational system, Arabic is the clear choice (in addition to Coptic being a "Christian" language as it is also the liturgical language of the Coptic Church; the Copts being a minority in Egypt by the point the language stopped being used on a daily basis).

This was compounded in many ways with 19th and 20th century colonial and nationalist policies and, with all due respect to a lot of people, the erasure of non-Arabic minority languages has as much to do with decolonial policies in the newly independent countries of the Levant and North Africa as it does with anything European colonizers did -- the idea of an Arab unity was hyped up as a way of transcending religious difference (this was one of the rationales behind Ba'athist ideology), which had the impact of hastening the decline of regional variant languages that weren't Arabic because being an Arabic speaker was a way to prove your loyalty and commitment to the state (a lot of the post-independence countries were authoritarian or something close to it, so appearing to be a loyal citizen was very important for your personal health and well being).

The Berber languages in North Africa, in particular suffered from not being the "prestige" language; even though Morocco and Algeria each now recognize a Berber language as official, they haven't exactly been pushing through programs to introduce education in Berber languages or media outlets, etc. The other issue is that Berber is a language family (similar to Romance languages) -- so, for example, the official Berber language in Morocco is Tamazight, which is mutually intelligible with only about 2/3 of the Berber languages spoken in Morocco -- if you're in that other 1/3, the "recognition" of Tamazight does pretty much nothing for you--and there are worries that this will hasten the decline of other Berber languages in favor of Tamazight.

That said, there are still substantial differences between the subregions. Moroccan Arabic (which has a heavy Berber and French influence) is very different from, say, Egyptian Arabic (which has both Coptic and Turkish influence), and both are quite different from Levantine Arabic, and the Arabic spoken in the Gulf (into which vocabulary from Persian and South Asian languages have been imported). This leads to regional competition and (sometimes not-that-) friendly rivalries about which dialect is better or closer to the original Arabic of the Qur'an, or which dialect is better in movies and song. (There is an artificial "modern standard Arabic" that was traditionally used in news media and literature, but it's not anyone's native dialect. For a long time, it was also the variant used in most Arabic as a foreign language classes, so ... people like me, for instance, would get off the plane in Cairo and sound like Shakespeare to everyone.)

Identities are also very slippery. People identify as Arab, but often it's one of many different identities they have, depending on who they're talking to. People often identify themselves as their nationality first (especially in North Africa); Arab often isn't the first answer -- it's one of several. Usually there will also be a region or village, and, depending on the area, a tribe or clan in the mix as well.

I think there’s some archived answers in the FAQ about the spread of Islam - I’m on a mobile device so it’s hard to look up and link. But the Mongols weren’t Buddhist! They were animist (some factions became Buddhist—and Muslim—after the conquests). And mainly motivated by profit — Marie Favreau’s incredible new book The Horde is really informative on the Mongols if you’re interested.