Let's first consider how tempting the idea might have bee, before we look at its opposing arguments. There were some military figures and advisors who thought they might have been somewhat useful. But there were many who thought that it was not clear that using nuclear weapons would have helped, or have been decisive in any way. Remember that the United States did use extensive conventional bombing against fixed Vietnamese targets in the North, and that didn't come close to ending the war. The use of tactical nuclear weapons was arguably not very useful in the guerilla warfare environment (tactical nuclear weapons were developed for use against large centralized encampments or tank columns or other things that the North Vietnamese tended not to use anyway). And if the weapons were used, but not decisive, then they would probably have been judged as a failed strategy. Even the full-on massacre of cities (e.g., like Hiroshima and Nagasaki) would not necessarily have caused the North Vietnamese leadership to end the war (separate from whether the atomic bombings of Japan actually did end World War II, one can say very clearly that the situation that North Vietnam was in during the war was nothing like the situation that Imperial Japan was in at the end of World War II, in terms of support, resupply, or threats of invasions from multiple sources).

Any use would have to be balanced against the costs. There were three costs explicitly considered at the time.

The first was escalation. Remember that the United States did not even try to invade North Vietnam during the war — they were merely trying to help the South keep control of its own country. Why weren't they more aggressive towards the North? Because they feared that if they did, it would easily spread into a wider war with China in particular. The politicians in particular (Kennedy, Johnson, Nixon) had no interest in escalating to that stage, and it was somewhat inconceivable to them that they could enter into a war with China without using nuclear weapons, and if that happened then the threat of serious retaliation against American allies or bases in the area (from China, who did not have an ICBM capability until the early 1980s) or against the USA itself (if the Soviet Union somehow got pulled into the conflict) would become unacceptably high.

The second, perhaps more subtle one, was the idea that if the US normalized tactical nuclear weapons use in a proxy war, they would come to expect the Soviets or Chinese to engage in such tactics in the future. The US way of doing war involves centralized bases — very tempting targets for a tactical nuke. There were scientists involved with advising the government (the JASON group) who emphasized that since US conventional superiority was so high, and its way of doing business involved heavy concentration of assets, then the US would suffer disproportionately if tactical nuclear weapons use was normalized — it would become an easy way for Soviet or Chinese proxy allies (like the Vietnamese) to "even the odds," if the Soviets or Chinese wanted to help them do so. Better, the JASON group suggested, not to go down that road.

The third, perhaps more overriding one (in the sense of explaining the behavior of the only people who could order nuclear use: the US presidents) are the political consequences. By the 1960s it had become a domestic and international norm that nuclear weapons were weapons that were politically unacceptable to use except in retaliatory situations (the "nuclear taboo" had been well established by that point). The US was already alienating its allies greatly with the war as it was; to (once again!) use nuclear weapons on an Asian country would result in massive protest, boycott, perhaps re-alignment globally. The short- and long-term political costs associated with breaking that "norm" were so high that the option seems to have been barely considered by the American presidents, even if the military experts sometimes toyed with the idea. Kennedy and Johnson appear to have had zero interest in nuclear weapons use at all in Vietnam. Nixon talked about the idea with Kissinger and made threats, but never came close to actually ordering nuclear use.

In short: The costs were perceived as potentially huge. The advantages were fairly slim if they existed at all. The three US presidents involved in the war, whether through calculated reason or gut instinct, appear to have never given it serious consideration.

For a detailed discussion of what we know about this and how we know it, see Nina Tannenwald, The Nuclear Taboo: The United States and the Non-Use of Nuclear Weapons Since 1945 (Cambridge University Press, 2007), chapter 6: "Nuclear weapons and the Vietnam War."