Essentially, the United States did not want a repeat of the Korean War. Invading North Vietnam would have placed US troops at a close proximity to China and in all likelihood would have provoked a reaction from the Chinese military. Additionally, the entry of China (and possibly the USSR) into the conflict would have zapped public support from the war effort.

The key objective was more or less to build up South Vietnam into a competent fighting force and secure their independence from North Vietnam. The unification of Vietnam under a US-friendly, united, Vietnam was of lesser importance and frankly a pipe dream. Turning South Vietnam into the equivalent of South Korean was the more realistic goal.

As for why the South did not win the war, there's probably not a single answer. Presidents as far back as Eisenhower acknowledged the popularity of Ho Chi Minh and the Viet Minh. Southern support for the communists would have been easily attainable even without Viet Cong columns filtering across the border. Vietnam had been at war with foreign aggressors for decades. Ergo, many in the South smartly viewed US intervention as motivated by self-interest rather than being in the interest of the ordinary Vietnamese citizen. US actions during the course of the war only bolstered these views.