As far back as the Eisenhower Administration, the US was rather concerned about a possible incursion into South Vietnam by the North. However, given the Korean War, many believed that a war in Vietnam would be similar with a large regular army invading the South. Towards the end of the Eisenhower administration and the beginning of Kennedy's term it became more obvious that internal subversion was a serious threat with the combined political (NLF - National Liberation Front) and military (PLAF - People's Liberation Armed Force).

Kennedy's intervention in Vietnam can't be discussed without also discussing Laos. The Eisenhower administration had advised military intervention in Laos to stop the Communist Pathet Lao. However, Kennedy chose to rather utilize negotiations with Moscow to end the fighting there. Kennedy felt that the land locked, dense terrain of Laos would not be conducive to US military operations.

As the NLF garnered more support in the countryside, Kennedy decided to increase troop strength to counter the threats. This more firm approach in part is explained by the need for his administration to appear strong after the Bay of Pigs. Regardless, Kennedy still believed that the main problem in Vietnam was political, and that the South's government would have to make a more concerted effort to solve the social problems in its country to combat the NLFs political power in the countryside.

With increased troop strength the US began a strategic hamlet program to separate the country side populace from the NLF. This was seen as a threat to the NLFs work in the countryside so far, however, strategies were designed to defeat it through outright assaults, and infiltration. As the NLF increased their attacks on strategic hamlets, US military advisers became more and more concerned calling for more troops to combat the threats.

Now President Johnson had to make the call as to whether escalate the war or to continue in an advisory position. While Johnson did not want to escalate, the continual attacks and large infiltration by Northern guerrillas caused him to increase US reprisals with air campaigns and military strikes without largely increasing US military equipment and manpower. Johnson hoped that the North would see his reluctance to continually escalate the war, and would agree to settle to terms of peace.

The North, however, was unwilling to talk peace on the US Government's terms and rejected negotiations that did not include the US's unconditional withdrawal from Vietnam.

You are right that Johnson was heavily focused on the Great Society Program in the US, and it plays heavily in his decision to escalate the war in Vietnam. Many Southern Conservatives in the US opposed the great society while also supporting increased military intervention in Vietnam. This support for increased military presence was also echoed by Johnson's own cabinet (many like Rusk, McNamara, and Bundy were hold overs from Kennedy), and numerous other foreign policy experts. If Johnson did not send more troops and suffered a defeat, he worried that his enemies would use it against him, and making him seem weak He worried that if this happened it could inhibit his domestic political goals. So in a way, agreeing to escalate helped assure the appeasement of political enemies to pass domestic reform.

This being said, Johnson, though hesitant to escalate ("It's damned easy to get in a war, but it's gonna be awfully hard to extricate yourself if you get in"), he completely rejected the idea of full American withdrawal. To quote George Herring:

"Johnson had made Vietnam a touchstone for his presidency, and could not bear the prospect of failure...Obsessed with appearing weak, he could not accept the idea of being president the first time 'we've ever turned tail and been shoved out of a place.' He did not like the fact that the United States was bogged down in Vietnam. But it was there, he concluded, and 'we've got to conduct ourselves like men.'"

Sources:

Duiker, William J. Sacred War: Nationalism and Revolution in a Divided Vietnam. Boston: McGraw Hill, 1995.

Herring, George C. America's Longest War: The United States and Vietnam, 1950-1975. 4th Ed. Boston: McGraw Hill, 2002.

US involvement in Vietnam is very complex and evolved over a 20 year period. I can help answer your question From a military perspective. Troop deployment to Vietnam represented a way for the US Army and Marine Corps to prove their worth in a time where the US strategic vision of war was characterized by an heavy reliance on nuclear weapons.

Both Truman and Eisenhower resisted direct intervention in Vietnam because they viewed it as peripheral to the "real conflict." Battle in Europe and a nuclear exchange with the Soviet Union. Eisenhower even resisted intervention to preserve French control (Johnson as a senator also spoke out against intervening in favor of the French at Dein Bien Phu).

At a strategic level, resources were diverted away from the Army and Marines to the Air Force and the Navy. Eisenhower pursued the "New Look" which was a policy that emphasized strategic nuclear weapons as a way to both deter potential conflict and win any resulting conflict. It was assumed that NATO ground troops would be simply unable to halt massive Soviet armored columns. Instead their role became to funnel the Soviet advance into pockets that could then be nuked.

The results of this policy on the Army and Marines was pretty clearly demonstrated in Korea, where the US Army was a shadow of the WWII juggernaut. In the age of jet aircraft, strategic bombers and ICBMs, Army leaders struggled to prove the military value of traditional infantry and armored formations.

Enter Maxwell Taylor. Taylor was a veteran of WWII and Korea and served as the Army Chief of Staff under Eisenhower. He was critical of the New Look policy and championed the relevance of the Army. Kennedy was also critical of Eisenhower's policies and brought Taylor back after the failure of the Bay of Pigs invasion. He later recalled Taylor from retirement and brought him on as a sort of special adviser bypassing the Joint Chiefs, the traditional military advisers to the President. Taylor re-entered on at a critical juncture, where the United States was considering the limits of supporting South Vietnam.

Taylor championed the idea of "brush fire wars." The logic was that the US Army could not resist the Soviets in Europe, where it was accepted nuclear weapons would have to be used. But instead the Army could be used to fight smaller low-level conflicts where nuclear weapons were necessary. Taylor championed a tradition of low-level interventions in places like South America. To him Vietnam was the perfect place to prove his ideas. In particular Taylor was insistent on a long term troop heavy commitment to Vietnam from the outset and helped steer the policy that caused the United States to be more involved in the early 60s.

Obviously Taylor was not the only person advising involvement, but he was a critical voice at a critical time and he had a lot of influence over Kennedy. I'm sorry I can't comment more on the diplomatic component but my knowledge is military focused.

My source on Taylor is from Thomas Ricks "The Generals." Ricks is highly critical of Taylor whom he views as misleading the President and worsening relations between the JCS and civilian leadership at a critical points. Unfortunately I am on the road at the moment and away from my copy, but I can look into it tomorrow when I get back if you'd like.

To add onto that, while he didn't start US involvement there and was largely left with the job of cleaning up the mess(in more ways that one: Vietnam directly led to the bill for the Imperial Presidency being due, which is what I'm getting into), Nixon definitely played a crucial role in America's Vietnam War. I think he could be added to the list: Nixon was far more interested in a host of other foreign policy issues (the USSR, Europe, Middle East, and especially China/East Asia) than who would rule in the long term in Saigon. Nixon saw Vietnam as a relative short term issue, in contrast to things like arms control or opening China, and like everything, Nixon was great when thinking 20 years down the road but absolutely horrible when thinking about next year. So often the pessimist, he was very overly optimistic on how quickly he could get a Korea-esque deal for Vietnam, and the US military paid for it in blood during in 1969. But like his predecessors, it dragged him in. Vietnam wrecked his Presidency much like it did Johnson's.

Haldeman and Kissinger were absolutely right: Vietnam and the domestic fallout related to it led directly to Watergate, far moreso than Nixon's obvious eccentricities and mental issues or the hatred he inspired, intense even on the standards of America's historically nasty, turbulent political scene. Vietnam was sort of like a cursed touch, like jungle quicksand when you think about it. Nothing good came of it. Although Reagan did an admittedly excellent job of giving America the ability to believe in itself again, to give it a pair again(I don't agree with everything Reagan did to this end, but I'll always give the Gipper full credit for that: it was very important), Brent Snowcroft once mentioned that it was only with Desert Storm that we fully managed to put to bed the battering to our foreign policy that Vietnam caused, the hatred, the division. More Democrats would eventually vote for the Second Iraq War than the First, keep in mind. So, I agree with him-it took the better part of two decades. And that was squandered in Iraq little more than a decade later.

There was a recent discussion of early US perceptions here. The Pentagon Papers remain an invaluable source, not least for having been compiled "from the inside". Until 1950 State seems to have been a voice of reason. Deepening engagement seems to have generated its own logic: "Was it the image it would give of the U.S. if a country fell to communism?" Increasingly, yes.