In 1946, only a year after he had become death, destroyer of worlds, Oppenheimer was asked in a closed Senate session about if it was possible a handful of people could smuggle an atomic bomb into New York City and blow it up.

Oppenheimer replied "of course", which was followed by

What instrument would you use to detect an atomic bomb hidden somewhere in a city?

to which Oppenhimer responded with a "screwdriver", in order to open every crate entering the city.

In other words, there was anxiety about portable nuclear weapons from the very start of the Cold War. Oppenheimer's comments were quoted in the 1951 novel The Smuggled Atomic Bomb which is roughly about the scenario mentioned in the question: physics student Duff Bogan discovers a plot to smuggle pieces of nuclear bombs into the United States piecemeal and assemble them while stateside.

The actual term tends to be "nuclear suitcase", referring to the idea of having a fully assembled portable bomb. They've been discussed more at length here and here by /u/Kochevnik81 and /u/restricteddata; despite persistent rumors, there didn't seem to be any from the Soviets that surfaced. A former Russian Security Council Secretary, Alexander Lebed, talked in the late 90s about a set of nuclear suitcase bombs, but none of the potential sites he mentioned in the US yielded anything.

However, we're discussing a slightly different scenario, more like the 1951 novel's idea. As the Hugh Sidey report is the only real source for this, it's worth quoting a little. Note Sidey did follow JFK closely and wrote one of the first biographies -- here's also a good oral history interview conducted by Sidey -- so it is unlikely he made the story up, but please note just because JFK said a rumor doesn't make it true. In the Cold War, conspiracy theories were like the air.

In late July 1961, President Kennedy, just back from the grim Vienna summit with Khrushchev, asked me to dinner in Palm Beach. After daiquiris and Frank Sinatra records on the patio, his three guests and I gathered around the table for fish-in-a-bag, a White House recipe. Between lusty bites, Kennedy told the story of Khrushchev's anger over West Berlin, the island of freedom in the Soviet empire's East Germany. "We have a bustling communist enclave just four blocks from the White House," I noted, meaning the Soviet embassy. Kennedy paused, fork between plate and mouth, and said, "You know, they have an atom bomb on the third floor of the embassy." Aware of JFK's love of spy stories, I said something like, "Sure, why not?"

Other than the one-shot nature of the story (which doesn't make it into the aforementioned biography) there are plenty of reasons to be doubtful. Embassies are absolutely hot when it comes to spying, and while the USSR was the undisputed champion of bugs, the US certainly made plenty of attempts, and it would be highly risky to assume any such discussion of nuclear weapon assembly would go unrecorded.

Furthermore, the Ms George Pullman Mansion (the building being referred to by JFK) dates to 1910 and was not really built with a nuclear lab in mind; here's a picture of the inside. The only major alterations were in 1933, with an enlargement by the architect Eugene Schoen, and an expanded wing done in 1977.

While these issues can technically be overcome, we essentially have a situation with no evidence to begin with, and offhand rumors about Soviet activities were so common there's no reason to suspect this one has special status.