Neither the US or China were particularly concerned with the possibility of nuclear war. When Deng visited the US at the end of January 1979, he informed Carter that he intended to attack Vietnam. In that conversation, Deng indicated that "It is not conceivable for the Soviets not to react at all. But we do not expect a large reaction." There's no mention of nuclear war, and because of the detail he is going to elsewhere, I would conclude that it isn't an oversight. Carter tried to dissuade Deng from invading, but the threat of nuclear warfare is not one of the many things he brought up either.

Chinese concerns over Soviet nuclear forces were directed towards an anticipated land war with the Soviet Union. The Soviet Union had significant military buildup close to the Chinese border, including tactical nukes, which shaped Chinese attitudes about the Soviet Union from the late 1960's until the mid 1980's. The PLA understood a direct conflict would likely be nuclear: they made preparations for "early war, major war, nuclear war", and the PLA played war games around a Soviet nuclear strike in 1981. It seems that Chinese leaders saw the Sino-Vietnamese War as just one of many proxy wars, like the Ogaden War or Angola, where the use of nuclear weapons would have been unprecedented and excessive.

Source: Xiaoming Zhang, Deng Xiaoping's Long War: The Military Conflict between China and Vietnam, 1979-1991