To answer your question, we have to start with the cause rather than the act. While it is difficult to establish just how often it happened, I feel like any discussion of atrocities in Vietnam has to be done out of the perspective of those who did it.

The Vietnam War was a counterinsurgency war. Now, counterinsurgency wars have an absolutely devastating effect on the human mind for the men who carry it out due to the fact that you are not fighting against a soldier in uniform but rather a guerrilla that really looks like any other peasant or urban worker out there. Since the concept of front lines as we know it (i.e. a set of precise line on a map that shows progress on the battlefield etc.) is very rare to find in counterinsurgencies, it makes it even more difficult to distinguish between civilian and foe. But for the American infantry soldier in this war, perhaps their lot was even worse than soldiers that have had to endure this kind of warfare before.

To really get to the bottom of this, I will now talk about the American strategy in Vietnam and how its choice of strategy inadvertently led to the rise of civilian casualties. When the US entered the Vietnam War, the choice of strategy was that of a conventional strategy. While pacification efforts and counterinsurgency strategies had been suggested to both the military and civilian leadership, it had been turned down. For the US Army, it was due to the fact that it went against every established doctrine they had. For the US government, it was seen as too slow. It would take too long to pacify and really gain the trust of the people, so instead they chose what they believed would lead to a quick victory: the complete annihilation of the enemy through attrition. So, the choice was then a strategy of attrition. This would then become the infamous "search and destroy".

But what really was search and destroy? As far as strategies goes, the name gave it all away. The concept was the following: The infantry soldier would go out in search of the enemy. When found, they would use their superior fire power provided by either air or artillery and annihilate them. That was the gist of it. Now this does sound fairly simple and in a layman's view of war, this is really all it would take. However, this works fine in a conventional situation but definitely not when you're facing an opponent using irregular tactics. If we look at an average patrol, we'll see the fact that the ordinary soldier very rarely encountered the enemy. If they did, it was very rarely they who found them but rather, it was the Vietnamese who found them first. But the enemy still seemed to be around at all times. Even during the firefights, many did not even see the enemy up close. The VC or NVA became a faceless caricature that didn't seem to want to fight face to face.

To put you in the mindset that this kind of warfare brought to an ordinary soldier, I'm going to walk you through this. Imagine that you're Private Stevens, a young grunt in the US Army. You're out on regular patrols, looking for an elusive enemy. As you approach the outskirts of a village, one of your fellow friends step on a hidden mine. Your first instinct is to obviously blame the villagers. They knew that the mine was here, didn't they? They wouldn't just allow someone to plant it here. A few days later, another friend is killed. This time by a sniper. You're constantly feeling anxious about running into an ambush, stepping on a mine or being killed in some other way. But at the same time, your frustration grows. Why won't these cowards show themselves? The hamlets in the area and the villagers - they have to know something, right? For all that you know, they're all VC. They're all the enemy. You heard stories back at the base about kids killing GI's with grenades and old women sticking knives into the back of young men. They must have some truth into them, right?

And so it begins. The civilians slowly but steady becomes the enemy in your eyes. They can't be trusted. They're too different from us, most would say. If they allow mines being planted in the outskirts of villages or nearby, aren't they all collectively responsible? The twisted thinking of a young man, afraid and angry, not to mentioned armed can in these situations be dangerous. It's the feeling of getting back at them, to get your revenge that fuels soldiers to commit atrocities and which led to plenty of civilian deaths in Vietnam. But these killings were not ordered. These were far more like a natural consequence of the strains of war on these young men.

However, the upper chain of command was not without any responsibility. You see, alongside search and destroy, there was one other controversial thing: The concept of body count. Since this wasn't a conventional war and since it technically wasn't a counterinsurgency war either, they couldn't really measure success in any other way but the bodies they found. A high body count meant that the US was winning. This genuinely turned the war into a war of numbers. For example, an officer that wanted a promotion or fill a quota set by the higher instances of command would sometimes exaggerate the numbers of kills or just plainly write in civilian casualties as enemy kills. So it wasn't directly encouraged or ordered from above in chain of command, but was instead something that grew out of human error. How morbid it might seem. A commanding officer could encourage killings of civilians by deeming a hamlet or a specific area before a patrol as VC area.

So, to sum it up: US forces did indeed intentionally kill Vietnamese civilians for reasons they thought were justified. These were not technically ordered by high command, but could through the use of body count be twisted even further. Abuses in the form of violence and threats against civilians were recurring but it was not systematic and did not differ from other counterinsurgency conflicts with widespread civilian abuse like the Algerian War.

Even counterinsurgency conflicts known for their civility and for the fact that they actually succeeded usually have atrocities very early on in the conflict before there was a big (or gradual) change towards proper counterinsurgency thinking. One such example is the Malayan Emergency (1948-1960) which is commonly seen as a textbook example of a successful counterinsurgency. In the very first year of the war, before a proper understanding of the conflict and how to combat it was reached (which wouldn't be until several years after), a platoon from the Scots Guards murdered 24 unarmed men in what was to be known as the Batang Kali massacre. While the Batang Kali massacre deserves a whole post for its own, it goes to show you that the nature of counterinsurgency to the untrained soldier can be very damaging and make you commit actions that in the case of Batang Kali more resembles something that German divisions tasked with anti-partisan duties committed during World War II.

For further reading on the topic of Vietnam and killing of civilians, I recommend Bernd Greiner's War Without Fronts: the USA in Vietnam.