You can roughly divide Vietnamese writing system into 3 periods:

Before 13th Century:

Vietnam (then Giao Chỉ) was conquered by Han Empire in 111BC. Chữ Nho (Chinese characters) were introduced and used in both official documents and literature works. Vietnam regained independence in 939AD, but Chinese was still adopted as for official purpose until early 20th Century.

13th Century to early 20th Century:

Although Chinese was still being used in formal writing, elite class started to use chữ Nôm. It had 2 portions: chữ Nho (standard Chinese characters) to represent native vocabulary (think Kanji in Japanese) and new characters locally created based on Chinese model. In this second portion, each word was represented by a single character, which was different from Japanese kana and Korean hangul.
During this period, literate people studied both: chữ Nho for national examination (to be appointed as an officer) and then chữ Nôm (writers and poets, especially women, since they were not allowed to attend national exam). Chữ Nôm was widely used in literature in this period, but it was never studied and classified systematically.

Latter half of 20th Century onward:

Chữ Quốc Ngữ (literally national language) became official.
However, it was transcribed and used as early as 17th Century by Western missionaries:

• French Jesuit missionary Alexandre de Rhodes was considered the one who contributed the most to the creation of modern Vietnamese writing system. He wrote Dictionarium Annamiticum Lusitanum et Latinum in 1651 after 12 years in Vietnam. (There is a street named after him in Hochiminh city).
• French Catholic priest Pigneau de Behaine wrote Dictionarium Anamitico-Latinum in 1773, which was modified and published by another missionary Jean-Louis Taberd in 1838.

Modern Vietnamese writing system didn’t change much after their contribution. However, not until the French invaded and colonized Vietnam in 19th Century that chữ Quốc Ngữ was recognized as official language. Officer Marie Gustave Hector Ohier signed a decree to replace chữ Nho by chữ Quốc Ngữ in official documents in 1869. Later, in 1879, it’s required to use chữ Quốc Ngữ in school, started from South Vietnam. Same thing happened in North Vietnam starting 1910.
So to answer your question: chữ Nôm was never studied and classified systematically, was never used in official documents, so it naturally died out. Chữ Nho was only used by officers and elite class, commoner never had a chance to study. Chữ Quốc Ngữ was forced by French government to be used in both official documents and education, so it started gaining traction and widely used.

Source: Vietnamese citizen, Vietnamese history textbook, some other Vietnamese history books (not translated) and some names recalled thank to Wiki.