The Prophet Muhammad did not appoint an official successor and so, following his death, the leading figures of the Meccan emigrants and Medinan society gathered to decide the matter of leadership. Regarding the matter of the Prophet not appointing a successor,

Al-Bazzar in his Musnad (a collection of traditions or hadith) states on the authority of Hudayfah (d. 656), that the Companions of the Prophet asked: "O Apostle of God! wilt thou not appoint a successor unto us?" He replied, "Verily, did I appoint a successor over you, and were you to rebel against the successor appointed by me, punishment would come upon you."

Jalaludin as-Suyuti (d. 1505), Tarikh al-khulafa (History of the Caliphs), p. 8

There are a handful of Shi'a sources claiming that Muhammad appointed his cousin and son-in-law, 'Ali b. Abi Talib, as his successor. This claim, however, has been largely rejected by early and Medieval Muslim historians as well as contemporary scholars.

Western historians largely reject as fabrications of history Shi'i claims that Muhammad appointed 'Ali as his successor and say there is no evidence that Muhammad appointed anyone as his successor.

Valerie J. Hoffman, The Essentials of Ibadi Islam, p. 6

Abu Bakr was eventually nominated to be the leader of a Muslim state by the joint Meccan/Medinan committee.

Abu Bakr was the natural choice. After Khadijah [first wife of the Prophet], he was the first person to accept Muhammad as a Prophet and convert to the new religion. He had been with Muhammad during his flight from Mecca. He was even appointed by Muhammad to lead the prayers in the Prophet's Mosque in the final days of his life.

Firas Alkhateeb, Lost Islamic History, p. 30

Furthermore,

Ibn Sa'ad has recorded on the authority of Hasan b. 'Ali, "We found that the Prophet had made Abu Bakr take the lead in public prayers. We approved, therefore, for our temporal affairs one whom the Apostle of God approved for our spiritual concerns. We thus gave precedence to Abu Bakr."

Jalaludin as-Suyuti (d. 1505), Tarikh al-khulafa (History of the Caliphs), p. 9

Before his death, Abu Bakr appointed 'Umar b. al-Khattab to be his successor after consulting with prominent members of the Islamic state.

The Companions [of the Prophet] consulted one another and said to him [Abu Bakr]: "We have decided to leave it [the matter of succession] to you, O Successor of the Messenger of God." He said: Give me time so that I may choose someone who will be most pleasing to God, most protective of His religion and His slaves." Then Abu Bakr called 'Abdur-Rahman b. 'Awf and asked him about 'Umar b. al-Khattab. He [ibn 'Awf] said: "By God, he is better than you think he is." Then Abu Bakr called 'Uthman b. 'Affan and asked him about 'Umar b. al-Khattab. 'Uthman said: "By God, what I know is that what he conceals is better than what he shows outwardly, and there is no one else like him among us." Abu Bakr said: "May God have mercy on you, by God, if I were not to choose him, I would not choose anyone but you." He [Abu Bakr] also consulted a number of the Companions concerning 'Umar and almost all of them were of one mind.

Mahmood Shakir (d. 2014), Tareekh al-Islami (History of Islam), p. 101

Abu Bakr appointed 'Umar as his successor as he felt that 'Umar would be effective in continuing to expand the borders whilst maintaining the teachings of the Prophet.

Whereas Abu Bakr appointed 'Umar as his successor after confirming his character and leadership abilities from certain individuals, 'Umar introduced a much more comprehensive method of appointment. He created the shura (advisory council), consisting of six prominent Companions of Muhammad. The six were 'Ali b. Abi Talib, 'Uthman b. 'Affan, Abdur-Rahman b. 'Awf, Sa'ad b. Abi Waqas, az-Zubayr b. al-'Awwan and Talhah b. Ubayd-Allah. To remove any suspicion of nepotism, 'Umar was very careful to exclude his relatives.

He commanded them [the shura] to assemble in a house and consult with one another [until they had appointed a successor]. 'Umar stipulated a [consultation] period of three days. He knew that consultation would not take place only among those six men, rather public opinion in Medina would also be taken into consideration. He set a limit of three days so that their choice would be based on the agreement of the majority of those present at that time in Medina.

'Ali Muhammad as-Sallabi, The Biography of 'Uthman b. 'Affan, p. 86

At the conclusion of this period of consultation, 'Uthman b. 'Affan was appointed the successor to 'Umar.

'Uthman was assassinated in 656 and did not appoint a successor. Being the senior-most Companion of the Prophet alive and his relative, 'Ali b. Abi Talib was the obvious choice and he was elected as the new Caliph by the community of Medina. His appointment, however, was met with significant opposition and eventually resulted in civil war.

As a comprehensive and concluding note,

When the Messenger of God was about to die, he appointed Abu Bakr as his representative to lead the prayers. People were, thus, content to accept him as caliph. No mention was made of royal authority, because royal authority was suspected of being worthless, and because at that time it was the prerogative of unbelievers and enemies of Islam. Abu Bakr then appointed 'Umar as his successor and 'Umar followed Abu Bakr's example and fought against the (foreign) nations. The caliphate, then, went to 'Uthman b. 'Affan and 'Ali. All these caliphs renounced royal authority and kept apart from its ways.

Ibn Khaldun (d. 1406), The Muqaddimah, p. 162