Comparison with Eisenhower is useful here.

At the start of 1953, after assuming office, Eisenhower worked on a plan with advisors to break through to North Korea with an overwhelming show of force, including nukes on the slate of options, and that using them "would be worth the cost". In a National Security Meeting of March it was agreed that "the taboo which surrounds the use of atomic weapons would have to be destroyed" and that "in the present state of world opinion, we could not use an A-bomb, we should make every effort now to dissipate this feeling".

Some past historiography of Eisenhower has tried to handwave some of this away; see, for example, Campbell Craig in 1998: "His strategy to evade nuclear war was to make American military policy so dangerous that his advisers would find it impossible to push Eisenhower toward war and away from compromise". However, Eisenhower (unlike Truman) wanted the National Security Council to be "developing consensus behind presidential decisions" and while Secretary of State Dulles was perhaps the main advocate for nuclear weapons -- he was the one who first brought up demolishing the nuclear taboo -- Eisenhower also reportedly said "the American public no longer distinguished between atomic and other weapons ... nor is there logically any distinction".

Perhaps fortunately, the real determination of Eisenhower was not tested. Stalin died on March 5 (before the "nuclear taboo" meeting) and unbeknownst to the US at the time, Stalin had been the advocate for continuing with North Korea, but the team that followed (including Khrushchev) felt the Korean War had done damage to communism and were looking for a way out. This led to an armistice in May which was signed in July.

Somewhere in late May, after the armistice had already been agreed to, Dulles (the prime nuclear backer) made a veiled comment about the war being expanded to Nehru of India intended for the Chinese. This comment, however, was never conveyed to the Chinese, although Dulles in 1956 gave a story in Life about how the nuclear threat (which was made after peace was essentially settled, and never even made it to Chinese ears) somehow stopped the Korean war in its tracks.

In reality, what happened was that Eisenhower had privately considered nukes, while publicly never needed to bring them up.

Now, regarding Nixon. Here he is, speaking to Haldeman in 1968:

I call it the Madman Theory, Bob. I want the North Vietnamese to believe I've reached the point where I might do anything to stop the war.

Certainly, in public there was the attempt to convey a willingness to escalate.

When Nixon took office in 1969 there was strong priority to end the Vietnam War as soon as possible, and a set of military plans known as Duck Hook were formulated (essentially, using heavy bombing and mining). The plan was eventually canceled by Nixon himself (sometime in early October, popular opinion was against and some of his advisors thought it wouldn't work) but there was still a "readiness drill" launched, a "series of increased alert measures designed to convey to the Soviets an increasing readiness by U.S. strategic forces" going for 17 days and ending on October 30. The Soviets seemed to think it was a bluff; a Soviet official regarding a similar 1973 alert stated "Mr. Nixon used to exaggerate his intentions regularly. He used alerts and leaks to do this" -- this no doubt is alluding to the prior alert in 1969.

A reporter at the time asked Kissinger about the use of nuclear weapons, and he stated outright "the policy of this administration not to use nuclear weapons" but still considered the possibility of "a nuclear device" in on a railroad pass. (So, a public "no, but....") One of Kissinger's aides mentioned regarding the "madman theory" that the North Vietnamese might genuinely believe a nuclear weapon was possible but "we wouldn't go out of our way to allay their fears about that".

Now, the overall question is how serious nukes were considered privately. There are two documents (released in 2005) from that time that mention nuclear weapons.

The first, a "Draft Memorandum on the Contingency Study" (that is, Duck Hook) which discusses what nature an "action" would take. One possibility, that it would be "brutal and sustainable", the other, that it would be "self-contained", and in particular, that the president "cannot, for example, confront the issue of using tactical nuclear weapons in the midst of the exercise. He must be prepared to play out whatever string necessary in this case."

The other, a list of "questions" includes:

To what limit of force should we be prepared to go in order to spur meaningful negotiations? Should we be prepared to use nuclear weapons?

However, this is the full limit of the hard documentation: raising questions without advocating. One can draw inferences from other private comments that perhaps the thought crossed Nixon's and Kissinger's minds, but compare with Eisenhower's NSC actively talking about dispelling the nuclear taboo. So while Eisenhower privately considered nuclear, and publicly did not mention them, Nixon did roughly the opposite, giving the public impression that he was just possibly a madman enough to use nukes, but without much internal serious discussion, even given Duck Hook, of actual use.

...

Burr, W. (2005). The Nixon Administration, the “Horror Strategy,” and the search for limited nuclear options, 1969–1972. Journal of Cold War Studies, 7(3), 34-78.

Burr, W., & Kimball, J. (2003). Nixon's nuclear ploy. Bulletin of the Atomic Scientists, 59(1), 28-73.

Burr, W., & Kimball, J. P. (Eds.). (2006). Nixon White House Considered Nuclear Options Against North Vietnam, Declassified Documents Reveal: Nuclear Weapons, the Vietnam War, and the "Nuclear Taboo". National Security Archive.

Jackson, M. G. (2005). Beyond Brinkmanship: Eisenhower, Nuclear War Fighting, and Korea, 1953-1968. Presidential Studies Quarterly, 35(1), 52–75. http://www.jstor.org/stable/27552659

Kimball, J. (2006). The Nixon Doctrine: A Saga of Misunderstanding. Presidential Studies Quarterly, 36(1), 59–74.