I think there are two problems at play here. First, is the estimates that get thrown around are often at their maximums. Second, is the a combination of demographic/chronology.

So, to start we have numbers from the Soviet government on the population of the Soviet Union during the time frame. Sadly we do not have a complete census before the Soviet Union that would be relevant. To start, the Soviets pegged their population during their first census in 1926 at 147 million. Now, things get dicey with numbers and historians have argued about the numbers since they were published. In 1936 the Soviet Union did another census. The census came back with 161 million originally. The results were tossed and the official numbers was 170 million. Finally, we have the last census that is relevant, in 1959. The number was 208 million.

Now looking past raw data and at the history. Russia suffered greatly through these years. But, not as much as we had originally thought. The idea that Stalin killed “20 million” is very outdated and not really the consensus among historians now. The rough estimate, which is probably the most accurate we can get at this point is between 6 - 9 million dead.^1 This includes the highest cost on human life, the Famine of 1932-1933 (also known as the holodomor). The “Great Terror” and collectivization and its discontents being the runner up until the Second World War. We can see the dip in expected population growth between 1926 and 1936, with Stalin ordering a “new” census with more “accurate” numbers.

However, hopefully the second problem is kind of coming into focus. Even with the terrible loss of life, the Soviet Union still grew. At the onset of Stalin, Russia had a population of 147 million and even after with the first census numbers, still grew to 161 million even after losing 6 - 9 million to Stalin’s regime. It endured, then endured even still through the Second World War with the losses between 13 - 31 million lives. It gained territory so the numbers are skewed, however the Russian State (within the USSR) still grew after the Second World War and 1959. The effects were felt, no doubt, it was massive death never before seen in Russia.

So to sum up. The Soviet Union did suffer, however its population endured because the original historical estimates were off, and additionally the amount of time during each catastrophic loss of life was spread out.

^1 Source for these numbers are from Timothy Snyders article Hitler vs. Stalin: Who Killed More? in NYBooks. The full article can be found here. The relevant quote is:

For the Soviets during the Stalin period, the analogous figures are approximately six million and nine million. These figures are of course subject to revision, but it is very unlikely that the consensus will change again as radically as it has since the opening of Eastern European archives in the 1990s.