That's Avignon, or actually the "Comtat Venaissin" surrounding Avignon, Carpentras, Orange, Venasque and some other towns. So the answer to "how did Avignon end up outside of France" is "it's pretty complicated!"

The reasons for this go all the way back to the way Charlemagne's empire was divided in the 9th century and the way those territories evolved afterwards. Avignon was in the Kingdom of Arles, which was not part of France, but was one of the constituent parts of the Holy Roman Empire. There were various other parts of the Kingdom of Arles, but the important ones here are the Margraviate of Provence (including the Venaissin and Avignon), the County of Provence (the city of Arles, Marseille, Nice, etc.) and the County of Forcalquier.

By the 13th century, the Margraviate of Provence was held Counts of Toulouse, and the County of Provence of the County of Forcalquier were ruled by a branch of the royal family of Aragon in Spain. Among all this territory, Avignon was big enough and wealthy enough to resist really belonging to anyone, so it was sort of ruled by its own citizens as a little city-state, but subject to all the other things happening around it.

Toulouse was, theoretically, always part of France, so the Counts of Toulouse owned territory that France considered part of France, and other territory that the HRE considered part of the HRE, but which was in reality pretty much totally independent. In the early 13th century, France and the Papacy waged the Albigensian Crusade against Toulouse, ostensibly to root out heretics, but it was always pretty obvious that France also wanted to exert greater control over the south (i.e. what is now "the south of France").

In the aftermath of the crusade, three important things happened: most of the Venaissin (except for Avignon) was annexed by the Papacy; it was arranged that Joanna, the daughter and only child of Raymond VII of Toulouse, would marry Alphonse, the brother of King Louis IX of France; and Louis and Alphonse's other brother, Charles, married into the Spanish ruling family of the County of Provence.

In 1249, Raymond died, and Toulouse passed to Joanna and Alphonse. They had no children and they both died within days of each other in 1271, so as far as Louis was concerned, Toulouse and Avignon and Provence and all the rest passed under royal control thereafter. But in fact Joanna's will bequeathed the Venaissin to the Papacy (which it already controlled anyway). The rest of Toulouse and Provence were gradually absorbed into France thanks to the advantageous marriages of Louis' brothers, but the little county of the Venaissin remained Papal territory.

Meanwhile, the Avignon itself hadn't passed to anyone in particular...until Charles conquered it in 1250, in his capacity as the Count of Provence, even though Avignon had previously not been in the County of Provence. Charles was a pretty ambitious guy and also ended up ruling the Kingdom of Sicily and Naples. So, somewhat confusingly, the ruler of Avignon was actually the King of Sicily.

In 1309, the Papal court moved to Avignon. The reasons for that are somewhat beyond the scope of your question, so I'll just note that since the popes already owned the Venaissin, and both France and Sicily were closely allied with the Papacy, it was a convenient place for them to live. This period is called the the "Avignon Papacy" (or the "Babylonian Captivity" depending on your point of view), and the popes remained there until 1377. During this period, Sicily sold Avignon to the Papacy in 1348, so the Popes ruled Avignon and the rest of the Venaissin directly thereafter. There is still an absolutely enormous fortress/palace in Avignon, the Palace of the Popes, dating from this time.

So, that is the story of how Avignon ended up as Papal territory. As for why it stayed outside of France for the next 400 years, that's no less complicated, but in brief, the Papacy never wanted to give/sell it back, and it was a bit difficult for France to get control of it the way they did in the 13th century - you can't arrange marriages with the Papacy! After the Papacy moved back to Rome, the county was socially and culturally French, and part of France in all but the strict legal sense. France did invade and occupy it several times, but legally the Venaissin remained papal territory all the way up to 1791, and in fact the Papacy did not officially give up their claim until 1814.

Most of the histories of Avignon and the Venaissin seem to be in French, but for a good recent history of the Avignon Papacy in English, see Joëlle Rollo-Koster, Avignon and Its Papacy, 1309–1417: Popes, Institutions, and Society (Rowman and Littlefield, 2015). Chapter 5 deals with how Avignon came to be a Papal territory.