There's no prohibition about foreign troops being on "Muslim soil" (soil, as an inanimate object, cannot be Muslim, regardless of what Usama bin Laden claimed). The idea that all of Saudi Arabia -- especially the eastern province, where coalition troops were stationed during the 1990-91 war -- is somehow sacred territory to Muslims is a fairly recent invention, and it's tied to the idea of the Saudi monarch as "guardian of the two holy places"--which are both in the Hijaz, hundreds of miles away.

The issue was specifically Mecca - only Muslims are allowed to enter the city (and, especially, the Grand Mosque). Mecca is surrounded by mountains, so the idea is that only Muslims are allowed to see the holy mosque (let's just set aside the issue of the Mecca webcams and documentaries for now). There are checkpoints on the roads into Mecca (religion is specified on visas issued to foreigners who visit Saudi Arabia), and there is a "non-Muslim bypass" that allows vehicles carrying non-Muslims to go around the city without entering the sacred precinct.

The optics of having foreign troops assist with the operation to retake the Grand Mosque itself was already bad enough--the incident was a huge embarrassment to the Saudi monarchy--without having them be non-Muslims, so they were converted en masse to make that part palatable to an already outraged public.

Yaroslav Trofimov's The Seige of Mecca tells this whole story in a relatively readable way, although I don't necessarily buy the subtitle's claim that the episode marked the "birth of al-Qaeda."