Any college-level course covering any aspect of modern US History will have an entire unit going in depth on the Vietnam war. It's a difficult topic, with a lot of moving parts.

The answer given by some commentators in the former Soviet Bloc, in China, and even in countries where a sizable section of the intelligentsia was Anti-American like France and Italy, is that by the mid-1960s the US military-industrial complex had grown so large that any excuse was good enough to start a protracted conflict. However, if you look into the source material and take into account the different backgrounds of Kennedy, Johnson, and Nixon, this answer is less than satisfactory.

In fact, there is a school of analysis that says the politicization of the conflict was so thorough that the military could have very easily won the conflict, if only military leaders had been given the resources they asked for and the politicians had limited themselves to setting clearer goals.

However, there's also an argument to be made that the United States, or the military of any great power, is in no way equipped to build a state and manage a counterinsurgency. Then again, the counterargument to this counterargument is that to win the conflict, the Việt Cộng had to conduct large-scale military offensives, which it wasn't equipped to win.

So, given that mind-numbing quagmire, why did the US stay for so long; and while we're at it, why did the United States get involved in the first place?

The most prominent figures pushing for intervention in Kennedy's cabinet, Secretary of Defense Robert McNamara and the Chairman of the Joint Chiefs of Staff General Maxwell Taylor, believed that the defeat of the government in Saigon as a consequence of what was (up to then) fundamentally a civil war would drag the rest of Southeast Asia into a wider conflict spanning the whole region. Remember that in this point in time, the US Government's policy towards communist regimes is best exemplified by the "Long Telegram," sent by George Keenan, the Chargé d'Affaires in the US Embassy in Moscow in the late 1940s; arguing that communist regimes believed themselves in natural conflict with capitalist ones. Although Keenan later stated he had been completely misunderstood (even affirming that almost every act of aggression by the Soviet Union had been triggered by the United States) the United States would believe itself to be in natural conflict with the Soviet Union and Communist countries until its collapse.

The flip side of this argument involves what Senator J. William Fulbright called, "The Arrogance of Power;" whereby Fulbright and other non-interventionists argued that the United States, which had been unscathed by war in the last century, and was seeped in the erroneous conviction that it could not lose a war. As a consequence of this conviction, politicians and military leaders made several fundamental miscalculations in assessing the conflict. However, there's also an argument to be made that the government knew it had its credibility at stake; if the US wasn't willing to support its allies in Vietnam, how could it expect other countries to honor alliances, or even want to enter them in the first place.

Hubris at all levels of the decision making-process coupled with a conviction that an escalation was necessary to "save face" is particularly important when evaluating Johnson's actions once he took over from Kennedy. Although there was no lack of people in the Cabinet and State Department that spelt out without mincing words the kind of difficult strugge that would ensue given an escalation of the War in Vietnam, Johnson did not back down.

Maybe some other president might have been able to objectively evaluate the overconfidence of the military, set clear war goals, and listen to the state department's warnings. But Johnson was not that president. He had been a congressman since the 1930s, and had witnessed the failure of the so-called congressional "China Lobby" to procure enough support for Chiang Kai-shek's Nationalist forces against Mao's Communists. Now that he was president, Johnson wasn't going to let another Asian country slip from the US' on his watch. Further, Johnson, as a provincial Texan, did not hide his disdain for the Washington establishment. He called them, "The Harvards;" and seems to have wanted to both impress and undermine them. Pushing forward in spite of their warnings and recommendations did just that. Lastly, given his personality, Johnson was clearly not the kind of person who wanted to go down in history as a president who had lost a war.

Once Johnson escalated, the war was beset by political failures which catalyzed military ones; Johnson's National Security advisor, McGeorge Bundy, and the aforementioned McNamara were particularly indecisive in setting goals for the war.

This is just an overview of the major political forces which rendered the conflict so protracted. There clearly is a lot more to say. If you're interested in the topic, the following reading materials could interest you:

Kahin, George McT., Intervention: How America Became Involved in Vietnam (1986)

More recent literature on the topic:

Logevall, F., Embers of War: The Fall of an Empire and the Making of America’s Vietnam (2012)