It seems peculiar that while the East Asian cultural sphere nations of China, Japan, and Korea escaped full-scale invasions resulting in complete subsumption by Europeans, Vietnam was fully colonized by the French. Were there any significant differences in Vietnam's governance and/or military that made the nation prone to colonization, and/or did colonial powers lack the will to colonize entire nations further north of Southeast Asia? I also acknowledge that Taiwan was partially colonized by the Dutch and the Spanish, but why was it still of relatively short duration and limited impact compared to the French colonization of Vietnam?
PART I:
Before I discuss French colonization more, I want to speak a bit about Việt Nam's place in Asian Studies. You mention that Việt Nam is a part of the East Asian cultural sphere, and that is interesting. Generally speaking, when talking about "Southeast Asian Studies", Việt Nam is included in that. Part of this is because of geography, but that isn't everything. Part of it does indeed have to do with culture. For example, the Vietic languages (including Vietnamese) are in the Austroasiatic language family, which includes Khmer (Cambodian) for example. Another part of it is that Việt Nam had a cultural identity prior to adopting Chinese ideologies wholesale during the Lê Dynasty (which was from 1463-1778 CE). Indeed, the region had been colonized by Chinese dynasties for about 1000 years (~100 BCE-~900 CE). But Nam C. Kim points out in The Origins of Ancient Vietnam that there still was indigenous identity and in fact this identity was a big issue for the Chinese during that colonization. It also was a major point of pride and political positioning when the Lý Dynasty (1009-1225 CE) came into power a few decades after the Chinese were defeated for good. I discuss this more here. It is fairly common to say Việt Nam was "Sinicized" while the rest of SE Asia (Philippines, Indonesia, Malaysia, Brunei, East Timor, Singapore, Myanmar, Thailand, Laos, and Cambodia) was "Indianized". I am not a fan of these characterizations because I think they are too limited in scope. It also makes it look like their cultures were given to them by "greater" societies, and that's really an old way of thinking about cultural exchange. However, I understand your idea and, regardless of some striking cultural differences and despite its distinctly different governmental structure and state ideologies, Việt Nam is included in Southeast Asian Studies. I just wanted to point out Việt Nam’s place in Asian Studies (as well as in international organizations such as ASEAN) as a Southeast Asian country.
I should also point out that at the time the Dutch and Spanish colonized Taiwan, it was not "Chinese" yet. In that sense it would not be considered “East Asian” because of its more “Southeast Asian” cultures. The indigenous population was the largest population there at that time and they are Austronesian-speakers, which is the dominant language family of Maritime Southeast Asia, Madagascar, and the Pacific Islands.
Việt Nam's colonization by France is structurally interesting. In fact, the only part of Việt Nam that was directly colonized by the French (as in French governmental structures and even settlers) was the southern part, Sài Gòn in particular. The French called this area Cochinchina, and they administered it directly. They split Việt Nam into two other areas as well: Annam in the central region, and Tonkin in the north. These two areas were indirectly ruled by the French as "protectorates", nominally still under the Nguyễn Dynasty, though the dynasty still answered to the French and of course followed French policy. This model was also used in Cambodia and in Laos. Sài Gòn, then, became the focal point of French culture in Việt Nam. It was the primary focus of the colonial period, and it was the area that the French were most determined not to lose later on during the Indochina Wars.
Why did Việt Nam succumb to total colonization instead of being subject to unfair treaties like China, Japan, and Korea? These kinds of questions are always really difficult to answer I think, but I will try to provide an answer that maybe can hint at how colonization played out.
Europe actually has a long history of being involved in Việt Nam. The first Europeans arrived on Iberian trading ships during the early 16th century, but it wasn't until the 17th that Jesuit Catholic Christians, who had escaped persecution in Tokugawa Japan, gained a real foothold. Among the new Christian evangelicals was the French Jesuit Alexandre de Rhodes, who operated in the Red River area for ten years (the Vietnamese governments were not very friendly to the Christians) before being expelled. He still evangelized further south. During this time, the French and the Portuguese developed a Latin-based writing system for transcribing Vietnamese language, and in fact de Rhodes published the first Portuguese-Latin-Vietnamese dictionary. This system is called quốc ngữ, and it became the preferred writing system during the anticolonial movements of the 20th century and is still the official system today. However, during classical Việt Nam and through the colonial period, the dominate writing was hán tự (Chinese writing) and chữ nôm, which was an indigenized writing system derived from Chinese stylistics. These were the systems that the missionaries actually worked in. Despite the efforts at suppression by the government, Christianity actually was relatively successful since the Vietnamese people were rather open to new forms of worship. Việt Nam was and is a pretty loosely religious area, a melting pot of sorts. There were perhaps up to 400,000 Catholic Vietnamese out of a population of up to 10 million total Vietnamese by the time of the Tây Sơn Uprising. This relative acceptance was confirmed even more when the French bishop, Pigneau de Behaine, spent much of his life evangelizing and helping Gia Long unify Việt Nam, and played a major role in the treaty between Gia Long and Louis XVI in 1787.
Unfortunately, as the Nguyễn Dynasty progressed after Gia Long, the Vietnamese government clamped down on Catholicism in its state policy of conservative Confucianism. This is unfortunate because of the situation that was happening in France. There was a cultural reawakening of Catholicism in France. In particular, propaganda was proliferated by the Missions etrangeres de Paris (MEP) from the 1830s onward about the missionaries suffering at the hands of the cruel dynasty. Vietnamese rulers were characterized as bloodthirsty and barbarous by the French. Further, in 1852 Napoleon III came to power and France was set on becoming a global imperial power. The persecution of Catholics in Việt Nam became a pretext for Napoleon III to attack.
Việt Nam was not blind to the goings on in nearby China. The court kept an eye on the Opium War, which resulted in the British colonizing Hong Kong in 1842. France received a concession in Shanghai in 1847, as did the Americans. That same year, the French actually attacked the Nguyễn at Dà Nẵng in order to free Catholic missionaries. The court decided to limit travel abroad for its subjects, as well as limiting foreign trade to specific ports. They were trying to keep the coastal colonization of China from coming right down into Việt Nam too. They pursued a similar policy as Tokugawa Japan in that they limited foreigners' abilities to arm rebels that could potentially overthrow the government with foreign aid.
The primary reason that France attacked southern Việt Nam in 1858 was simple prestige and pride. During the period of French aggression, the idea of the colonizing mission was prevalent. The French felt that the "strong" and "white" should rule over the "weak" and "colored", and thus France was entitled to this land. France adopted a strong interventionalist foreign policy under Napoleon III from 1852-1871, getting involved in a new level of colonization all over the world. This also included involvement in the Second Opium War in China. France wanted to catch up in Asia to the other Western countries. Napoleon III was looking for any pretext to attack Việt Nam, and Catholicism was the most obvious one. And in 1857, Emperor Tự Đức executed two Spanish missionaries, providing the necessary pretext. After aiding the British in Guangzhou that year in the Second Opium War, the French attacked Việt Nam with help from the Spanish based in their colony in the Philippines. In 1858, the French and Spanish attacked and occupied Dà Nẵng, and in 1859 they attacked Sài Gòn.