Ganondorf himself claimed that the Gerudo being forced into a harsh desert was the reason that he betrayed and deposed the King of Hyrule and effectively put Princess Zelda into exile.
He seems to just be some weirdo with a lust for power. What's the historical consensus?
The earliest accounts of settlement in Hyrule by modern Hylians speak of a red-haired "wizard" people whom were likely slowly pushed out of Hyrule and into Gerudo by subsequent waves of Hylian migration and territorial expansion. While evidence of forced mass exodus or genocide is lacking, there does appear to be evidence of early conflict among the nomadic Hylians and Gerudo.
The modern historical consensus among Hylian academics is that Ganondorf's Invasion of Hyrule was based on a faulty casus belli with a weak claim to the throne of Hyrule; the coronation of the King of Hyrule was recognized by all inhabiting peoples of Hyrule.
Souces:
[1] (http://en.m.wikipedia.org/wiki/The_Legend_of_Zelda:_A_Link_to_the_Past) The Legend of Zelda: A Link to the Past - 1991, SNES.
[2] (http://en.m.wikipedia.org/wiki/Ocarina_of_time) The Legend of Zelda: Ocarina of Time - 1998, N64.
[3] (http://en.m.wikipedia.org/wiki/The_Legend_of_Zelda:_Twilight_Princess) The Legend of Zelda: Twilight Princess - 2006, GameCube.
Ganondorf was raised by two terrorists who fostered a deep resentment of the royal family. Some interpretations suggest that they groomed him specifically for the coup. His perceived "lust for power" may simply have been the result of the propaganda he was exposed to throughout his childhood.
So you could argue that the Gerdos' exile was indirectly responsible, in that it gave rise to the organization behind Ganondorf.
Despite their stereotypical reputation as thieves, tensions between the Hylians and the Gerudo were surprisingly minimal before Ganondorf's invasion. As we all know, the unique aspects of Gerudo genetics result in a largely female populace. Because they rely on Hylian men to bring forth the next generation (a fact well documented by one of the royal family's Shiekah stones), the continued survival of the culture is a testament of the healthy relationship between the Gerudo and Hyrule. Clearly, the Gerudo choose to live in the desert. Perhaps Ganondorf acted the way he did because of his status as a Gerudo holy figure. As the one male Gerudo, he was likely far too important to be given the opportunity to assimilate into Hylian society like many historians believe some Gerudo women did.
Souces:
1 The Legend of Zelda: Ocarina of Time - 1998, N64.
2 The Legend of Zelda: Twilight Princess - 2006, GameCube.