On the US side, the argument against the PTBT was that it would limit the ability of the US to fully test weapons at full yield, and would limit US work in high yield areas. That the Soviets had detonated several weapons in the "very high yield" area (e.g. the Tsar Bomba) meant that in that area, it was "ahead" of the US.

(The arguments against this argument were that the US had gotten better at underground testing, and that the future wasn't in arbitrarily high yields but relatively modest yields. The US also had a program in place where if the Soviets violated the PTBT the US would immediately test atmospheric weapons that would give it the "lacking" knowledge in high-yield areas. This "backup" program lasted until the 1970s.)

It should be remembered as well that underground testing is actually much more difficult than atmospheric testing in many ways, from a technical perspective. The US and Russia both got very good at it over the rest of the Cold War, but it was initially feared that it would severely limit the kinds of weapons and scientific developments a nuclear-armed nation would be able to accomplish. Indeed, that was part of its point, beyond the fallout concerns — to slow down the arms race.

The arguments from nuclear states like France and China that did not sign the PTBT was that it constrained their sovereignty and their ability to develop and test their military arsenals as they saw fit. This (sovereignty issues) was also why they did not sign the NPT until the 1990s.

If you include the CBTB in "other test ban treaties," the reason that many nations have either not signed or signed-but-not-ratified the treaty is because even if they are not presently planning on conducting nuclear tests, they don't want to constrain themselves in the future, in case — for whatever reason, but including maintenance on increasingly aging stockpiles — they need to.