It has to be pointed out that the book cited by /u/Nate20ASU does not promote the consensus view of the Vietnam War but instead puts forward what many analysts consider to be a myth of the Vietnam War. Summers, a colonel during the Vietnam War, argues that the United States could have won the war if it had declared war on North Vietnam and destroyed the North Vietnamese Army (NVA). As /u/Nate20ASU notes, Summers considers the Vietcong to have been a "symptom" of the NVA (the work cited here was published in the mid 1980s, well before the modern consensus on Vietnam coalesced, it should be noted). Some modern historians, however, such as Robert Brigham, argue that the war was not winnable at all in part because Americans in general (i.e., both Lyndon Johnson who escalated the war and analysts like Summers who supposedly present a contradictory view to Johnson) mistakenly believed that the Vietcong's efforts to drive out what they perceived to be imperialists was really an arm of the North Vietnamese efforts to conquer South Vietnam. Brigham concludes that the United States never had a chance to win the war because it misunderstood the enemy's nationalism as a machination of communism (i.e., that Hanoi controlled the Vietcong entirely), because it fought in a terrain that favored this nationalistic enemy (i.e., a jungle that they knew well), and because the United States never had the option of waging a full-blown war because of Vietnam's proximity to China (as /u/Nate20ASU notes).

While these are not the only two analysts of the war, it should be noted that Summers' On Strategy particularly appealed to soldiers and others who believed that the United States should have won the war and that Johnson, Nixon, and other leaders screwed the army out of a deserved win. Most of modern scholarship posits this as an incorrect, nationalist perspective. There is, however, a true scholarly work that promotes this view: Mark Moyar's Triumph Forsaken, which was supposed to be part one of a two part series, but which drew so much attention and criticism that it became a major work in itself and Moyar has yet to write the second volume, which was, ironically I'd say, supposed to be the major book out of the two.

There isn't a "main reason."

The problem with the Vietnam War was never a question about outright military strength. If that was the case the Vietnam War would have ended rather quickly, and probably at an incredible cost. Though I don't necessarily agree with Keith Taylor (a rather well known revisionist scholar), the US were indeed winning every strategic defense within Vietnam. One could argue that Operation Rolling Thunder actually achieved its goals, which was to destroy the infrastructure and production capacity of North Vietnam.

The problem however was that the US could never invade North Vietnam from 1961-1969 mainly due to the direct backing and stance of China which officially made clear that China would not turn away from a provocation. In other words, invading past the agreed parallel in the Geneva Accords would have essentially resulted in WWIII. Likewise, most of the Aid sent to North Vietnam was by China until the late 60s. Much of the aid came in the form of munitions, anti-aircraft crews, engineers, road-builders, and vehicles. The majority of that equipment thanks to the rather pragmatic dealings of the North Vietnamese at the time came from China despite both China and the Soviet Union post Sino-Soviet Split vying for the attention of the DRV.

The other problem occurred after the Tet Offensive in 1968. Though the Tet undoubtedly resulted in a loss for the Viet Cong, the United States couldn't offer a counteroffensive for fear of an even larger escalation of the war. Strategically the Tet Offensive and the resulting attempts to reclaim central Vietnam brought to light a slew of rather unpleasant happenings to the American public. Of which included the My Lai Massacre and the general lack of cooperation of the South Vietnamese which had been a continual problem since the Ngo Dinh Diem and after his assassination in 1964.

I would suggest that a good way of trying to understand why the US failed to win the Vietnam War would be to view the US as not the primary actor, but the patron and benefactor of the South Vietnamese. Though Orthodox scholars have a habit of describing the South as corrupt and incompetent for good reason, the opposite spectrum of the Vietnam War scholarship tends to describe the US as limited by both its domestic policy and the military having "one hand tied behind their back."

What both schools tend to miss are the political complexities that affected both North and South. Though the North had more stable leadership, despite a centralized government it too faced serious issues especially during the planning of the Tet Offensive. Likewise, the spy networks that were set up to infiltrate the North by the CIA were woefully inadequate in comparison to the Soviet trained Northern Spies. Despite all this, the standard of living and quality of life was abundantly better in the South. During the 1954 open border season, more than 2 million North Vietnamese fled to the South causing an enormous refugee crisis. The answer to why the US didn't win can't be answered by merely viewing the military and tactical mistakes. The distinct regionalism combined with international geopolitics only created more obstacles along with the more overt lack of cooperation within the South. In tandem with what seemed to be an unclear goal for victory and a very successful insurgency via the Viet Cong, the US's original thoughts of a Conventional War soon gave way to conflict that required the US to understand not only the regional politics of both sides, but also understand the culture and people of a nation and area of which the US had no idea about.

In hindsight Vietnamization and the escalation via the bombing of Cambodia by Nixon proved an abject failure. If there was one constant failure that I would point out by the US during the Vietnam War, it would be the paradoxically detached but intense involvement the US had within the RVN's internal politics and military. Confronted with political inertia and a fear of escalation that echoed the Korean War (as well as war fatigue from the American populace which had witnessed nearly 20 years of warfare), politically speaking the Vietnam War by 1973 was no longer viable. Even if the US had decided to stay, the progressive transition towards Vietnamization would not have stemmed the huge hole left by the rapid and alarming reduction of aid towards South Vietnam after 1973.

In short I would suggest that the US couldn't win because the war wasn't a war for the United States. The goal in the end was unification, and the support of the RVN (Republic of Vietnam) by the United States shouldn't be viewed as a war that the US fought for the US as much as it was an extended and heavily involved aid process for South Vietnam. Combined with a lack of understanding of the intricacies of the war and battlefield that continued to be propagated through the AVRN's high command (who were all US trained), victory became a longer and longer prospect.

PS: I should include a lot more primary sources, but outside of a few key notes from journalists within the South Vietnamese government, North Vietnamese sources are particularly difficult to parse, especially since most of them are in Hanoi or Da Lat and generally in Vietnamese and censored.

Sources:

1. K.W. Taylor, “How I Began to teach about the Vietnam War,” Michigan Quarterly Review, 43, no.4 (Fall 2004)

2. Edward Miller, “The Taylor-Buzzanco Debate and How we Think about the Vietnam War,” Journal of Vietnamese Studies, Vol. 1.

3. Robert D. Schulzinger, The End of the Vietnam War, 1973-1976, Nixon in the World: American Foreign Relations, 1969-1977

4. Lien-Hang T. Nguyen, “Waging War on All Fronts: Nixon, Kissinger and the Vietnam War, 1969-1972,” in Nixon in the World: American Foreign Relations

5. Merle Pribbenow, “General Vo Nguyen Giap and the Mysterious Evolution of the Plan for the 1968 Tet Offensive,” Journal of Vietnamese Studies, Vol. 3 No.2

6. Gary Hess, “The Tet Offensive: A Decisive American Victory or a Prolongation of Stalemate,” in Gary R. Hess, Vietnam: Explaining America’s Lost War

Many fantastic answers so far. Here is my contribution out of a purely counterinsurgency perspective.

To answer your question about why the U.S. was fearful of becoming aggressive in North Vietnam has much to do with the fact that the United States was scared of a repeat of the events in the Korean War. When MacArthur advanced to close the Yalu River during the war, China eventually gave pursuit of the United States. One the issues in Vietnam was again, the fact that North Vietnam bordered China, so the United States played this safely during much of the war.

To answer your question as to why the U.S. lost the war, there's a variety of different reasons. One reason is the beginning strategy enacted by Secretary of Defense, Robert McNamara, who believed in what was called the "body count" strategy. Essentially these were missions designed to count the number of North Vietnamese killed in the war. The problem with this however, is the fact that one, it's impossible to know how many enemies were killed by a napalm bomb, and two, soldiers would inflate the number killed. Another issue as to why the U.S. lost, was the fact that they tried to fight a war of attrition. This was a strategy used by commander, William Westmoreland, which eventually led to a stalemate in the Vietnam War. Finally, and perhaps the biggest issue in war, was the fact that the U.S. fought mostly a symptom of the war. This symptom of course was the Viet Cong, the guerrilla fighters in the war. These were the soldiers we are more accustomed to seeing in modern depictions of the Vietnam War such as the movie "We Were Soldiers", which depicts the Ia Drang Offensive. These guerrila fighters were a symptom of the war, that the United States dedicated much of the war effort towards, which made the U.S. take the strategic defensive. Adding onto these strategic failures by the U.S. forces, Vietnam had no infrastructure making it impossible to destroy supply routes, or caravans as the military did in World War II. All of these issues added onto why the U.S. lost.

Perhaps the best book I've read for this is "On Strategy" by Harry Summers. Summer's goes into great detail about how issues at home, North Vietnamese nationalism, and how strategic failures all caused the loss in Vietnam.