It was a combination of things:

1. Treatment upon returning home. Part of the mental healing process of killing is gaining affirmation for your actions. Police and military personal who are made to feel like they committed crimes by the general public can internalize those feelings contributing to PTSD.

2. Splitting up units. The norm for wars before Vietnam was units deploying together. Taken to its extreme, you had units composed of the young men from single communities. If a particular battle wiped out the unit, entire villages would basically die off from there being no young men left. Even when that wasn't the case, the Xth Infantry Division, or whatever, would all train and deploy together. While that was still the case in Vietnam, it was much more common for service members to deploy as individuals. It increases the sense of isolation and delays the formation of bonds between the service members. The U.S. military stopped doing that after Vietnam and now, for the most part, entire units rotate in and out of theater together.

3. More effective training. Prior to Vietnam, there is some controversial evidence that far fewer riflemen actively participated in combat than was previously thought. Western militaries recognized this "problem" and devised new training methods that reached their peak in Vietnam. In other words, there were far more individual "killers" in the form of riflemen on the battlefields of Vietnam whereas in previous wars, most deaths were caused by crew served weapons whose team element help protect against the feelings of guilt that often lead to mental health issues. Even today, few crew served weapons operators get PTSD compared to riflemen.

4. Less isolation after war. Most societies have recognized war as needing unique considerations. Many veterans in historical wars were thought to need a symbolic cleansing ritual before returning to their communities. In many previous American wars, this ritual took the form of the long, transoceanic boat rides. The men who fought and killed together were given several weeks to come to terms with what they did in a sort of safe space before they went back to their families. For many Vietnam veterans, they came back to the US by plane, sometimes within hours of last seeing combat. Lacking the time for reflection, they were immediately put back into society where they often had to come to terms with their actions in isolation from their fellow veterans while hiding their pain from their children and families. This is still an issue, but barring bringing back the long boat rides or isolating veterans for a month after their deployment, this is a problem that isn't going to go away.

5. More recognition and pathologization of the effects of war. PTSD has always been known and has gone by many different names. The "shell shock" of WWI is now easily recognized as that war's version of the modern PTSD. In other words, while there was an uptick in mental health issues from Vietnam, there was also more of an institutionalized understanding of the problem, which led to more general awareness by the public.

I'm sure there were other issues that contributed, but these are five of the big ones.

Source: On Killing by Dave Grossman. A lot of the data he uses is not considered entirely reliable but the overall points of the book are, in my opinion, sound.