This is a BIG question encompassing hundreds of years of history and numerous sects, all with their own theological outlook, but in essence it boils down to the question of who or what is a legitimate source of religious authority and by extension where socio-political authority stems from in Islamic societies.

I'll limit my answer to discussing traditional, mainstream Sunni Islam and Twelver Shia Islam. Note that other Islamic sects and groups have beliefs which vary considerably from what I've written below.

In very general terms -

For believing Muslims the question of the succession isn't just an historical argument, it directly impacts the way they practice today - to use a crude analogy it has some similarities with Christian arguments over Papal authority versus Biblical Scriptural authority. Although theologically the differences between Sunni and Shia are less stark than those between Protestants and Catholics.

Sunni Muslims believe that religious authority comes directly from the Koran and from the example of the life of Mohammed (pbh). They do credit Sunni religious scholars and leaders with having some authority to interpret scripture, but in a much more limited capacity than that ascribed to their Shia counterparts by Shia Muslims.

Sunni Muslims recognise Abu Bakr as having been properly elected to succeed Mohammed (pbh) as a political and social leader. He is described as having been a close companion of Mohammed (pbh) during his life and as a model Muslim. His suitability to succeed, in Sunni belief, stems primarily from his ability and willingness to emulate Mohammed (pbh) and they assert that Mohammed (pbh) chose him as his successor.

Shia Muslims believe that human beings other than Mohammed can be a source of considerable legitimate religious authority, especially if they have some claim to kinship with Mohammed (pbh). They don't believe in any Prophets after Mohammed (pbh) but they recognise certain individuals as having the gift of divine revelation.

This also impacts their interpretation of Islamic scripture as they place more emphasis on those Sunnah and Hadiths which are attributed to Mohammed's close family and associates.

They view Ali as having been the legitimate successor not only because they assert he was chosen by Mohammed (pbh) but also because they believe that he and his descendent were divinely appointed by Allah.

The most prominent manifestation of this distinction in modern times is probably the role of the Ayatollahs in Shia Iran, they exercise considerably more influence as interpreters of Islamic scripture and the perceived will of Allah than any equivalent Sunni clerics.

Further reading

http://www.cfr.org/peace-conflict-and-human-rights/sunni-shia-divide/p33176#!/

http://www.bbc.co.uk/religion/religions/islam/subdivisions/sunnishia_1.shtml

http://www.npr.org/sections/parallels/2007/02/12/7332087/the-origins-of-the-shiite-sunni-split

I'll be a bit late to the party, and u/intangible-tangerine already answered the question well, but there's a few things I need to add.

1. Broadly speaking, we can divide the authority of Prophet Muhammad into two spheres. These spheres generally overlapped a lot in his lifetime. These two spheres were his authority as revelator and ultimate interpreter of the Qur'an and his authority as the political leader or "sheikh", if you will, of the fledgling Muslim neo-tribe (this term was used by Reza Aslan in his book No god but God to describe the Medinan state, and I am immensely fond of it) based on common religious identity rather than tribal affiliation. Upon Muhammad's death, these two spheres of authority were up in the air. All self-professed Muslims (except the Ahmadiyya and perhaps other fringe groups) believe that Muhammad was the final prophet, and that revelation ended with him.

2. Upon Muhammad's death, the Muslim community found itself divided about the question of authority. I won't delve too deeply into the political machinations at first, but the split was largely political. Abu Bakr, Umar, and Ali were the three contenders for the succession. It seems that Abu Bakr and Umar met with other tribal leaders in the Muslim community, and at this meeting Umar pledged his allegiance to Abu Bakr, as did many other leaders. Shi'a sources indicate that this clandestine meeting, known as the Saqifa Bani Saidah (named after the tent of the tribe to which it belonged, where the meeting took place) took place while Ali was attending Muhammad's funeral.

3. Abu Bakr inherited Muhammad's temporal authority as sheikh or leader of the Muslim neo-tribe. His role, as Khalifat RasulAllah, or the Deputy/Successor of the Messenger of God, was largely political.

4. A significant minority of Muslims believed that the entire Saqifa was a sham, because according to them Muhammad had already appointed Ali as his successor at the Ghadir Khumm sermon during his pilgrimage - according to several traditions attested by both Sunnis and Shi'is, Muhammad called Ali up to a makeshift stage and proclaimed "man kuntu Mawla, fa haza Aliyyun Mawla" - he whose Mawla I am, Ali also his is mawla. The meaning of the word Mawla is debated - Sunnis insist that Muhammad was proclaiming Ali as his friend and ally (something Muhammad was fond of doing, as it seems that he and Ali were very close being first cousins and Muhammad having been raised by Ali's father Abu Talib as a son of his own), Shi'is claiming this was Muhammad's appointment of Ali as not only his successor in the sphere of political authority as the Khalifat RasulAllah of the Muslim neo-tribe, but ALSO as the successor to his spiritual authority without the status of Revelator. In other words - Muhammad was still the final prophet, but not the final interpreter of the Qur'an. In Shi'a eyes, it was Ali's job, after Muhammad's passing, to continue interpreting the Qur'an for the Muslims.

5. So to answer the question... WHY does it matter? Well, it matters differently depending on which side you fall. To Sunnis, things shook out more or less as they were supposed to - Muhammad's temporal authority was passed down properly for at least the Rashidun Caliphate. From the perspective of Shi'as, however, the leadership of the Muslim community has almost always been in the wrong hands (unless you are an Nizari or Bohra Ismaili Shi'a and accept the Fatimid Caliphs). The entire history of the Shi'a has been the denial of rightful leadership to Ali and his descendants. To them, it is more than a political betrayal - remember, Ali in Shi'a eyes, is also the interpreter of the Qur'an. However, this interpretership did not die with Ali - it continued in Ali's sons, and grandsons, down the male line. Ali's role become generally more mystified and exalted and by the time of his great grandson Muhammad al-Baqir and great-great grandson Ja'far as-Sadiq became fully fledged as the role of Imamah as all the major branches of Shi'a understand it today. Not only has the Muslim community been led astray politically, but to the Shi'a, it has also been led astray spiritually.

Certainly not discouraging any historically-relevant answers, but you would probably get a much better answer to this question in /r/islam/, especially since the differences between Sunni and Shia, as well as the other minority branches, is still very much an on-going issue in the Muslim world