During Operation Big Switch, 3,597 American POW were returned while 23 chose to stay. That works out to about half a percent of the American POWs deciding not to return to America. How Operation Big Switch worked was that POWs could decide to repatriate or not. People who chose to defect were held for 90 days to give them a chance to change their mind. 2 Americans that initially refused repatriation did exactly that during the "cool down" period. On return, they were both court-martialed in short order and were sentenced to 10 and 20 years in prison.

In interviews immediately after their repatriation, they both gave (in my opinion) fairly half hearted answers. Seeing as they were about to face a court-martial, we can assume they were trying to come off well. One meekly said that he had spent so much time reading and listening to things about how great China was, he initially thought he'd stay, then changed his mind. The other said something semi-coherent about wanting to go to China to avenge American soldiers who had died in the war and in the POW camps.

As for the 21 who stayed after the cool off period and actually went to China, their reasons for defecting varied. For some of them, as far as I can find, the only record of their reasons for defecting are specifically produced propaganda videos. These videos consist largely of political platitudes about peace, freedom, equality, escaping McCarthyism, embracing the virtues of Chinese-style Communism, etc. As far as actually recording their reasons, I don't consider those videos alone to be terribly compelling. Since they were recorded during or after the 90 day period, they weren't recorded under threat of torture, but that doesn't mean they weren't in any way scripted or coerced. I think it's easier to look at what people said later, when they were neither in China nor (presumably) under threat of court-martial.^1

Then again, what was said by the ones who eventually returned to the US is also potentially suspect. While none of them were facing court-martial^1, they were potentially facing hostile public opinion and the perception of being treasonous traitors, turn coats, commies, Reds, etc. So, I'll talk about the two cases that have the most documentation.

The example that seems somewhat typical is that of Samuel David Hawkins. He spent 4 years in China before returning to America, and was interviewed by foreign journalists during his stay, on television by Mike Wallace after his return in 1957, was covered in 21 Stayed: The Story of the American GI's Who Chose Communist China—Who They Were and Why They Stayed, and covered again the the 2006 documentary They Chose China.

When asked by Mike Wallace in 1957 why he defected, he said:

Hawkins: Personally I went to China to compare what I'd heard with what they actually practice in China. I went because I was very curious by nature.

Wallace: What had you heard?

Hawkins: Well, only what our Chinese captures had told us. How great they were working towards Socialism and the great strides they were taking in bettering the life of the Chinese people who for so many thousands of years had lived a life of oppression under various governments.

Wallace: Why did you feel this sense of identification, of wanting to know about them? Had you not been told by our own army?

Hawkins: No.

Wallace: They'd never discussed the Chinese with you?

Hawkins: No.

Wallace: They never told you what to expect were you to be captured?

Hawkins: Yes, they said we'd be killed.

There's another defector in particular with a very clearly articulated reason for defecting to China, Clarence Adams. Both from interviews in They Chose China and from his own memoir An American Dream: The Life of an African American Soldier and POW Who Spent Twelve Years in Communist China, he makes it clear that a large motivation for defecting was escaping the racism of the American south. He joined the military after someone in a group of black men he was with punched a white man in Memphis. When the police came to his house, he high tailed it to a recruitment office.

I'll note here that his story is often used in a "Look! American racism was so bad he fled to communist China!" way, but that's not the whole story. He was also a High School drop out who excelled at the indoctrination education given in the camp. He did so well that they gave him extra classes and let him lecture to other POWs in the camp. I point this out to say that there were things above and beyond simple racism that motivated his decision.

On the subject of "brain washing", there was a real concern about that in 1954, hyped up by various people. I don't know anything about the MK Ultra time line, but I do know that by 1956, the fears about fiendish Communist brain washing techniques was pretty well disproven with the publication of the Congressional report Communist Interrogation, Indoctrination, and Exploitation of American Military and Civilian Prisoners. It basically concluded that, while the Chinese and Koreans were pretty good at extracting information, they didn't have any kind of "new and remarkable techniques of psychological manipulation." That's not to say the report put a big enough dent in the idea that the public laughed off the publication of The Manchurian Candidate 3 years later. Simply that people in the government should have known better in 1956.

They Chose China by Shui-Bo Wang is free to stream online. It has a lot of interesting archival footage, as well as David Hawkins actually visiting China with the director, and interviews with various widows and family members. It's under an hour long, and watching it a while back sparked my interest in the topic. I highly recommend it, though it does present a particular bias.

1: As near as I can tell, none of the 21 were actually court-martialed, simply given dishonorable discharges (though Hawkins eventually got his changed to "other than honorable".) That's not to say that the government never attempted to go after any of them on their return. The House of Un-American Activities tried to go after Clarence Adams, but beyond that I don't have good information.