The USSR and Japan actually did fight against each other several times. The battles of Lake Khasan in 1938 and Khalkin Gol 1939were a part of wider Soviet-Japanese border conflicts in the region. Both also resulted in Japanese defeats, although with comparable Red Army losses as well.

Imperial Japanese colonial doctrine had two possible areas of further expansion. First was the Northern expansion, towards Siberia and adding to the lands of Imperial Japan’s puppet state Manchukuo. (Supported mainly by elements in the Imperial Japanese Army) The second was the Southward expansion, towards the nations of South East Asia and Pacific Islands (supported by the Imperial Japanese Navy)

The recent defeats of the Soviet Japanese border conflicts, however, in the end convinced the Imperial Japan that a southward expansion was more feasible. Events culminated in the signing of the Soviet-Japanese Non-Aggression Pact in April 13, 1941 to secure Manchukuo’s borders from possible future hostilities with the Soviet Union. Ironically it came at an opportune time as the USSR could now avoid a full on two front war and focus its full efforts in fighting against Nazi Germany and the other signatories of the Tripartite Pact.

Japan and the Soviet Union actually did fight several times in the 1930s, culminating in the Battle of Khalkhin Gol in eastern Mongolia in the spring and summer of 1939. A series of skirmishes and battles ultimately resulted in Soviet Forces under General Georgy Zhukov (the same man who would later become Marshal Zhukov and hold almost unparalleled respect in the Soviet Union) outmaneuvered Japanese forces, destroying several units and showing some of the weaknesses of Japanese ground forces. This defeat occurred simultaneously with political events on the other side of the USSR.

Stalin knew of the coming partition of Poland, but despite the German-Soviet treaty to split Poland, Stalin never fully trusted Hitler. He also knew that Japan still held expansionist ambitions into Siberia. This led to a real risk of the Soviet Union having to split its forces not just between two fronts, but to handle logistics between two fronts some 4000 miles apart. The defeat of Japanese forces at Khalkhin Gol allowed the Soviets to push a neutrality pact with Japan, which was signed in mid-September, just before the Soviet invasion of eastern Poland. As a result, Stalin could concentrate on the war in Poland and, eventually, against Germany without having to worry about fighting the Soviets. Similarly, Japan didn't have to worry about fighting the Soviets and could concentrate on the war in the Pacific. Both sides could fight their major foes without worrying about additional fronts sapping their resources.

Japan took the neutrality pact seriously, avoiding even interfering in much of the shipping from the US to the Soviet Union in support of the war in Europe. The neutrality pact survived most of the war, coming to an end when Moscow withdrew from it and declared war on Japan after the atomic bombing of Hiroshima and Nagasaki. Soviet forces took significant land from Japanese forces, including most of the Kuril Island chain, the ownership of part of which Japan disputes to this day.

For the Japanese, they're in a position where they desperately need a neutral mediator with the United States, Britain, and China. By 1942, Japan has already subsumed more territory than it can feasibly control, and Japanese leaders had understood that they could not win a long-term war with the United States. Instead, the best bet was to seize as much as territory as possible and then wear them down until a negotiated peace could be arrived at. Better to do that with a possible third-party.

People have posed the counter-factual of what might have happened had Japan invaded in 1941 when the Soviet Union was at its most vulnerable, but Japan was already bogged down in a war in China that tied down hundreds of thousands of troops, and it had been bested by the Soviet Union in the 1939 border conflict with the Soviets. Also, its own relations were worsening with the U.S., and splitting its energies was simply too dangerous.

For the USSR, the answer is easy: Nazi Germany. Early on, when the Soviets are fighting for survival and there's no obvious sign that the war will end in their favor, opening a second front was unthinkable. The Soviets face close to 90% of the German war machine, and until 1944 there are only a few side theaters tying down German troops to alleviate pressure on the Eastern Front. Sending troops to start another war would have been irresponsible in the extreme, especially given that Stalin was already smarting because his allies moved slowly (from his perspective at least) to help him with their own troops. It's not until Yalta (February of 1945) that Roosevelt gets a firm commitment from Stalin that he will invade Japan.