The political situation in Japan in 1941 simply did not support an invasion of the Soviet Union. Japanese politicians and military planners were divided on whether going for a Northern Strategy (invading the Soviet Union) or a Southern Strategy (occupying Southeast Asia/Pacific). Japan in the 1930s had lost a series of border battles against the Soviet Union and this had dissuaded many of its leaders form taking the northern strategy. This was further compounded by the rise of US naval power in the east and the collapse of relations with them following the invasion of china and the US oil embargo on japan. Many Japanese leaders sought to improve relations with the USSR in hopes that it would join Axis, leading to the signing of a Non-Aggression Pact between the two powers in April 1941.
Militarily japan was also not ready to go to war with the Soviet Union. It was deeply invested in a war with china which was pulling in massive amounts of men and resource, so by 1941 the army was overstretched in china and not prepared to open a new front against the Soviets. They were further dissuaded by the loss of frontier battles against the Soviets such as the Battle of Khalkhin Gol, showing the Japanese lacked the heavy weaponry to fight toe to toe with them. Japans strength came from its naval power which could not have been used effectively against the Soviet Union, planning to use it in Southeast Asia/Pacific where it wold be far more effective.
Thanks to the US embargo on Japan it had lost almost 80% of its oil production which would mean that any further expansion would have to satisfy Japans need for oil. While it was known that there was oil in Siberia it was difficult to reach, very isolated and difficult to extract and transport back to Japan. Compared to in Southeast Asia where there were large oil reserves that could be easily reached and transported by Japans large merchant fleet. This convinced both the Army and the navy that they should abandon the Northern Strategy and go with the Southern Strategy.