They are referred to as the Rashidun by Sunni Muslims (about 90% of Muslims). Twelver Shi'ites would use some very different language for the first three.

The idea is that the four Caliphs came to power through a justified way and ruled according to the texts of Islam. They all were of the first generation of Islam, those who lived with the Prophet. The fifth caliph, Muawiyyah, came to power in controversial circumstances and his decision to appoint his son caliph after him was seen as introducing a hereditary kingship to what had previously been a position chosen by the people. He is considered the first Muslim king since the caliphs after him were all Umayyad (hence the term Umayyad dynasty). After him, the companions of the Prophet had passed on so no one else was eligible for the title of "rightly guided" as no future ruler could eclipse the rank of a person who had been a companion.

Of course, you do have people who have been labeled as the "fifth rightly guided" caliph, generally Umar ibn Abdul Aziz, but it's done as a sign of praise. He's never actually counted amongst the Rashidun.