I responded to a similar question recently: What is the city in Southern France that was never taken by the French till the revolution? And why did it remain independent for so long?

Basically for a brief summary of that, the land around Avignon, the Comtat Venaissin, was bequeathed to the Papacy in the will of Joanna of Toulouse in 1271. Avignon itself ended up being ruled by the Kings of Sicily, who sold it to the Papacy n 1348.

/u/y_sengaku has previously answered Why did the Pope move to Avignon?