The prevalence of Shi'a Islam in Iran and parts of Iraq goes back to the foundation of the Safavid state in the early 16th century under Shah Ismail. Ismail came to power largely through the support of the Qizilbash, a group of semi-nomadic Turcoman peoples that tended towards more radical, millenarian Shi'a beliefs -Ghuluww. At the start, Ismail and his followers went so far as to claim that he was the mahdi, a prophesied redeemer figure who would return at the end of time. Early on, however, Shah Ismail ran into the Ottomans under Selim the Grim, who wasn’t too keen on Safavid expansion to the west and eventually the two sides met at Chaldiran in 1514 where Shah Ismail was soundly defeated. Afterwards, Ismail moved his power base deeper into what is today modern Iran, and Iraq became not only the territorial boundary between the two empires (though land occasionally changed hand in subsequent conflicts), but the ideological one as well.

After the defeat at Chaldiran, Ismail toned down the millenarian claims and the ideological position of the state moved towards the more moderate Twelver Shi'ism, though rulers still saw themselves as the representatives or anticipators of the Hidden Imam and sought to integrate the Shi’a faith into their political and cultural patronage. Prior to the Safavids, Iran was majority Sunni, and Persian scholars of Islam were actually quite popular in the Ottoman Empire, but subsequent rulers began to import Shi’a religious authorities from traditional centers of Shi’a thought, and gradually Safavid Iran became a center for Shi’a learning. By the time of the Safavid collapse in 1722, the Shi’a ulama, or judges, had become an increasingly powerful group in society, and in the vacuum left by the Safavids, were able to assert themselves as the representatives of the Hidden Imam (citing weak state authority) and were therefore qualified to lead the people.

So, to sum things up, early Safavid legitimacy was built on Ismail’s millenarian movement that saw the formation of the Safavid state. Post-state formation, the Safavid rulers tended towards a more moderate Twelver Shiism and sought to integrate their ideological position into the cultural life of the empire. Today, the Shi’a populations in Iran and parts of Iraq roughly correspond to the areas once under Safavid rule.

Dont forget Azerbaijan. About 85% of the people are Shia. This is due to Qara Qoyunlu khanate which were shia. QQ also ruled over most of today's iraq, eastern turkey (explaining the alevi-qizilbash, whom I belong to) and al-hasa (the only province of arabia with majority shia population). QQ was subjugated by Aq Qoyunlu.

It might be worth a few laughs to mention that Qara Qoyunlu means "black sheep" and Aq Qoyunlu means "White sheep"

As a follow up, if Mesopotamia was an Ottoman territory until the 20th century, how did the southern Iraqi Arabs become Shias?

To address one of the questions in your title that seems to have gone unanswered: The Safavids laid the foundation for the modern unified Iranian state and an identity based on a combination of Twelver faith and Persian language (in spite of themselves and much of the population being Turkic). The modern borders are definitely a part of the Safavid legacy, and therefore tangentially related to religious identity, even though Sunnis and Shi'a are to be found on both sides of both the Western and Eastern borders.