The nuclear arms race didn't start with weapons in the thousands of warheads; it started quite modestly. The US took decades to build up a substantial atomic force, and to build up the means of delivering them to targets deep in the USSR. The Soviet Union, in turn, spent decades just "catching up" to the US, both in terms of raw numbers, but especially in terms of delivery. The US had allies ringing the USSR, in which it could install military bases and bombers that could be used to attack the Soviets in the event of war. The Soviets had basically only one ally near the US — Cuba — and its one attempt to put nuclear missiles there had pretty disastrous consequences. The Soviets did not, until the early 1960s, have a credible way of attacking the USA directly — they were just too far away, and it is too easy to shoot down bombers. In the early 1960s the USSR starting deploying long-range ballistic missiles, which changed the situation, but prior to that its main method of deterrence was threatening US allies, like Western Europe, by proxy.

So both arms races started relatively modest but then got pretty out of control. Why, how? Part of that is just momentum — when you build a system that can churn out thousands of warheads per year, it becomes the default position to just do that — but there are other trends as well. One is that many of the weapons they were creating, by the late-1950s, were "tactical" weapons, not "strategic" ones. They were relatively low-yield (though beware that what counts as "low-yield" shifts dramatically in this time period — some of those "low-yield, tactical" weapons were the same strength as the Hiroshima bomb), and meant to be used against tanks, bombers, fighters, submarines, boats, etc., rather than cities, bunkers, and so on. These were developed in part because it was felt that the prospect of "we both commit genocide, or we get invaded" was not seen as a totally credible threat (it would be very hard to decide to "push the button" over a small incursion). Having some sort of half-measure ("we will still start to go nuclear, and it could escalate, but we'd try to stop that") was seen as making the nuclear threats more credible. So in the US case, maybe half of their warhead growth in the Cold War ended up being tactical.

The other issue is that while people today tend to imagine "one target, one nuke" is a sensible approach, this isn't how war planners thought about it then (or even now, frankly). They look at the odds of a weapon system malfunctioning, the odds of a weapon system missing, and the value of the target, and come up with an equation that tells them how many redundant nukes you need to aim at the target to guarantee its destruction to some high level of probability. For different weapons systems and targets, that can add up to quite a lot. Is this overkill? Almost certainly — because they were making very conservative assumptions about what would be destroyed, and wanted to err on the side of "caution" (destruction). But it also means that for every target you have, you end up multiplying the number of weapons. And if you are imagining a full nuclear war meant to totally destroy the enemy's capability to make war, then you can also find a lot of targets. Once they were in the era of "nuclear plenty" (thousands of new nukes per year) it became easy to justify having lots of them.

Lastly, I would just note that one self-enlarging aspect of the arms race is that many nukes are targeting other nukes ("counter-force" targeting). We knew where Soviet ICBMs are (you can see ICBMs with satellites and spy planes), and they knew where ours were. Would our ICBMs be able to destroy their ICBMs before they got off the ground, or vice versa? Probably not — the ICBM systems were designed with this obvious form of attack in mind — but both US and Soviet war planners couldn't quite imagine not trying to disable the opposing side's nukes (maybe you'll get lucky!), and so the consequence is that the more nukes you build, the more nukes the other side is going to build to target your nukes, and so on.

What stops this cycle are things like arms control treaties that put hard caps on the number of launchers, which allows you to say, "I can live with having X number of weapons as long as you keep yours to the same number," and put artificial, agreed-upon limits on the number of weapons. This is what the START Treaty (and New START) did. Even that doesn't really stop the "race," because you can innovate in other areas — the US has done lots of works on making their weapons more accurate, for example, which means you can allocate fewer weapons per target and still get that high chance of destruction, which gives you the capability to threaten more targets with the same number of missiles, and so on.

Lastly, it is worth noting that generally speaking, the people who chose to make these deployments often did not believe that they had the capacity to "destroy the world." They tended to believe that nuclear war was, to some degree, survivable and fightable. Sure, maybe you'd lose 60% of your population — but you can still do a lot with 40%, and in a few decades maybe you could get back to the original level of prosperity! Such things are not quite as baseless as they seem (because nukes are not quite as powerful as people tend to imagine them to be), but there is also a bit of madness to them.

Absolutely lastly, I just want to also note that even those who didn't think nuclear war was survivable tended to subscribe to the logic that nukes prevented their own use if you made them scary enough. That is, it wasn't the prospect of nuclear war they were looking forward to, it was rather that they thought that with enough nukes, the Soviets wouldn't dare try to attack us. In this mindset, more nukes = more terror = more deterrence. It can also be pointed out that should deterrence fail (which was completely possible, and almost happened several times), then they also equate to "more destruction, more death," on a truly global scale. Such is the bitter paradox of nuclear deterrence: "The worse things get, the better they are."