You piqued my interest, so I went and looked it up. It was my understanding that even though the liberation of French prisoners (including many from other French colonies) was complicated it was pretty much done by 1956.

I found this article in the Revue historique des armées (French journal on military history) which does a pretty good survey of the issue and seems to agree with me. I'll summarize it.

In october 1954, France freed 65,000 Vietnamese prisoners. Vietnam only freed 11,000 out of an estimated 37,000 kept in more than 100 camps, often in mountainous regions. The prisoners were routinely subjected to hard labor and political reeducation.

After a thorough review by an ad hoc half-French half-Vietnamese UN committee it was estimated most prisoners kept by the Vietnamese died (the French association of former POWs in Indochina estimates an average of 60% of French POWs died in the camps). The number of French POWs still under custody of the Vietnamese army was revised to 6,450.

A lot of those seem to have ending up deserting, including legionnaires who were allowed to return to their country of origin when it was in the Soviet bloc and Vietnamese auxiliaries who got politically "converted". Some also managed to escape. Still, in 1956, that is two years after the end of the war, French intelligence reported that there was probably still about 50 French (European) intelligence and commando troops being kept prisoners by the Vietnamese. The Vietnamese considered these troops to be war criminals and it is possible they may have kept them but I can't find any further info on that.

Anyway it seems to me that most POWs were freed by the Vietnamese and that those who disappeared did so because they deserted or because they died due to the harsh conditions in which they were kept. I certainly cannot find sources arguing that a significant number of POWs were kept till the 80s. Except for one case! I did find a press article about the French-Vietnamese aide to general de Lattre de Tassigny, Huynh Ba Xuan, who was kept imprisoned by the Vietnamese for 23 years. He was not freed until May, 1976 when he lived in hiding for a while and finally came to France in 1980.

Source is Ivan Cadeau, « 1954-1956, le départ du corps expéditionnaire français d’Extrême-Orient », Revue historique des armées, 258 | 2010, 67-81.