It's a counterfactual. We can speculate, and there may be documents that suggest how they might have or probably would have acted, but we necessarily cannot know simply because that did not in fact happen.

What you're asking is, Would the USSR have gone to nuclear war if the US had invaded Cuba. Although all we can do is speculate, in the absence of evidence, I think the odds are pretty strong that the Soviets would have swallowed hard, blustered loudly and threatened to attack US ships at sea, but gone no further. That was the problem with the Cuban Missile Crisis--it was hard to come up with half-measures. It was all or nothing, nuclear war or nothing. Khrushchev was a blustering bully, but he wasn't reckless. And as I've heard the story told, Russian radars were white with images of US nuclear bombers at the borders of the Soviet Union. The nuclear devastation of the USSR was inevitable.