There were people within the US government and esp. the US military who very much supported the idea of using nuclear weapons to defend the islands of Quemoy and Matsu from a major Chinese assault. Not just publicly (where such things as sometimes "floated" for political/rhetorical means) but privately, at meetings of the Joint Chiefs of Staff and the like. Eisenhower himself, at this time, was not at all opposed to the idea that nuclear weapons could be used tactically and in a limited way (e.g., he did not regard this as a deterrence situation). Internal, top-secret policy papers from the time make it clear that Eisenhower and his advisors largely regarded atomic bombs as "munitions" in the same category as conventional arms.

So it is pretty reasonable to assume that this was a pretty seriously entertained idea. The tricky thing is, it was tied to other ideas about the US response, and these all hinged on reactions to whatever the Chinese did. So they weren't used, but this wasn't because someone said, definitively, "we aren't going to use them," or even "we are going to use them" — it's because the circumstances didn't actually end up going down the path where the "use" question would have been resolved one way or another. But on the whole I think it is pretty fair to say that the possibility of their use was taken very seriously and that with slightly different events they probably would have been used.

As for the Chinese, they appear to have thought the US was serious about the use of them, and that this would have engaged them in a larger conflict than they were willing to be involved with at that time. They do not seem to have necessarily regarded this as an "end of the world" scenario — they are a large country with a lot of people and the number of character of nuclear weapons at that time were still fairly limited — but as a larger entanglement than two islands were worth. However Mao also resolved, at that point, to make sure that China developed its own nuclear arsenal, so it could not longer be put in a one-sided situation of "nuclear blackmail" of this sort. This was the origins of the Chinese atomic bomb program, as a result.

On the Quemoy and Matsu crisis, see Gordon H. Chang, "To the Nuclear Brink: Eisenhower, Dulles, and the Quemoy-Matsu Crisis," International Securit 12, no. 4 (Spring 1988), 96-123. See also esp. Nina Tannenwald, The Nuclear Taboo, chapter 5. On the Chinese reactions to the crisis and the nuclear threats, see Lewis and Litai, China Builds the Bomb, chapter 2.