I understand that some sects/religions such as Alawites are considered to be Muslim by some Muslims and not by others. It seems like this might pose a strong practical challenge: if they are, they should presumably be encouraged to go on hajj but if they are not, they should presumably be barred from Mecca.
Has Mecca ever been closed to any sect/religion that claimed to be Muslim based on a ruling that they were not really Muslim?
Firstly, Alawites historically did not perform the hajj. Indeed, Alawites historically did not perform many rituals that are considered basic Islamic practices by both Sunnis and mainstream Shi'is, including fasting during Ramadan and praying at mosques. Only within the last century have there been systematic attempts to bring Alawites more into the Islamic mainstream, particularly by identifying them with Twelver Shi'is. Naturally, this effort has been promoted by Iran.
As for your question, there have been periods in Islamic history where one state or another has barred certain sects from performing the hajj, though this is definitely the exception rather than the rule. The Sunni Ottomans and Shi'i Safavids were geopolitical and religious rivals; the Ottomans prevented Iranians from performing the hajj^1 until the Peace of Amasya in May 21, 1555,^2 the first peace treaty between the Ottomans and Safavids. In the text of the treaty, the Ottomans also demanded that Shi'is stop the cursing of the first three caliphs, a practice called sabb. Later treaties between the two empires repeated these clauses.
When the First Saudi State controlled Mecca after the second capture of the city in 1805 from the Ottomans (the Saudis briefly occupied it in 1803), they began a policy of prohibiting pilgrims from performing the hajj - this is because they saw most other Muslims at the time as outside the fold of Islam, especially Shi'is and Sufis. Even before their capture of the city in 1801, the Wahhabis killed several hundred Iranian pilgrims in a raid on a caravan. The Saudis later allowed Sunnis to perform the pilgrimage under certain restrictions, but the city was recaptured by the Ottomans in 1812.
"From Rhetoric of War to Realities of Peace" by Ernest Tucker in Iran and the World in the Safavid Age, p. 85.
Sabri Ateş, Ottoman-Iranian Borderlands, p. 20. Wikipedia gives the the date as May 29 but this (scholarly) source gives it as May 21.