FWIW, here is my earlier answer to a similar question.

TL;DR: Stalin did in fact believe that Germany was the biggest threat to the Soviet Union, and the war was all but inevitable. However when exactly it would begin was unknown. The intelligence wasn't nearly as obvious as it may seem without the benefit of hindsight and the Soviets had reasons to distrust the British (who were obviously not a disinterested party). Lastly Stalin didn't want to jump the gun and hasten the war by declaring a premature mobilization of the Red Army.

Stalin did not expect Hitler to keep to the agreement indefinitely, what was a surprise (to him at least) was that Hitler attacked so soon, while the British were still in the fight. The Germans were trying to avoid a two-front war, hence the Molotov-Ribbentrop pact in the first place. From Stalin's perspective, it appears that it would then be somewhat implausible for the Germans to launch their attack on the USSR while their flank was exposed to the British. What was not counted on in his thinking here is the idea that it was not truly going to be a two-front war, assuming the Germans could force capitulation within the first year.

In terms of why he disbelieved his spies and warnings from the west, that's harder to answer. Stalin trusted the British about as far as he could throw them, and he assumed (correctly) that the British would make efforts to attempt to convince the Soviets to intervene. His assumption that British or American intelligence on the impending German attack was thus fabricated was not correct. Hitler reportedly gave assurances regarding the increase in forces on or near their border, which could not possibly be concealed on such a grand scale, saying that he was shifting forces to protect them from bombing and scouting as part of a buildup to an invasion of the British Isles. In hindsight this excuse is completely ridiculous, but if Stalin already believed that it was unlikely that Hitler would attempt to take on the USSR and UK simultaneously, and if he beloved that he was being deliberately fed bad intelligence to try and pull him into a war he didn't want to get involved in, then perhaps it is not completely out of character for him to think in this instance that the Germans were legitimately not about to attack. I could perhaps speculate that part of it is that it was what Stalin wanted to believe, that the Germans and the UK would be locked in a long term struggle and the Soviet Union would gather strength unmolested on their periphery. But unfortunately we do not have full access to Stalin's real thoughts on this matter, we can only attempt to infer from his actions, the information he had at his disposal, and so on.

As others have said, Stalin knew that war was going to happen eventually. What he assumed, though, was that Hitler wasn't crazy enough to open a second front against the Soviets (at least not at the outset). He thought that the Germans would drive westward and only turn eastward later, to avoid a two-front war. In fact, at the outset of the Molotov-Ribbentrop pact, he was probably very confident in the Soviet position - he had averted war for a time with the Nazi's, who he predicted would drive westward and weaken capitalist states, while at the same time taking back territory in eastern Europe lost over the past couple decades during war and revolution.

He also didn't expect the Germans to advance so quickly. The swift surrender of France was alarming to Stalin and threw into question his entire strategy towards the Germans; if the west is conquered too quickly, then the Nazi's would turn eastward much sooner. So even before the front was opened he was already worrying about impending war. The Soviets accelerated their plans for "Sovietization" of the Baltic states. Further, the disaster of the Winter War in Finland not only showed the poor state of the Soviet military, but also was coming close to a German conflict, as the Nazi's drove eastward through Norway, something he obviously wanted to avoid.

In November 1940, in a meeting with Molotov, Hitler invited the USSR to become a partner in the Tripartite Pact, effectively offering them to join the axis. This of course wasn't a reality, given that both sides knew war between them was coming, but both wanted to buy time. Both sides negotiated on how to come to an agreement, but ultimately Hitler approved the invasion of the USSR in December. During this entire period the Soviet leadership was working to get the military in a war-ready state.

The Germans attacked in June 1941. The night before, a German sergeant defected and warned the Soviets, who weren't sure if he was telling the truth or a provocateur. Despite the request of his generals to make the troops combat ready, Stalin denied that an invasion was imminent and asserted that it was possibly the latter.

Stalin was informed shortly after 3am that the Germans had invaded. Military leadership arrived to meet with him shortly thereafter, at which he continued to express doubt about the invasion, positing that it could merely be a provocation by German generals and that "Hitler surely doesn't know about this." He sent Molotov to meet with the German ambassador and refused to order a counterstrike until he returned.

Molotov returned. His meeting with the Ambassador did not go well (obviously). The German claim was that they had to take countermeasures in response to the aggressive military build-up on the Soviet's side. They had clearly not needed much of an excuse to invade. In response to Molotov's account of the meeting, Zhukov claims Stalin "silently dropped into his chair and became immersed in thought. A long and painful pause ensued." He concluded that a counterattack was acceptable as long as the troops didn't violate German borders. This was four hours after the invasion commenced. Zhukov continues:

During the first day he was not able to really take himself in hand and get a firm grip on events. The shock to I.V. Stalin caused by the enemy invasion was so strong that his voice even became softer and his instructions on organizing the military effort were not always appropriate to the situation."

Chadaev later recalled:

Early on the morning of 22 June I caught sight of Stalin in the corridor. He had arrived at work after a brief sleep. He looked tired, worn out, and sad. His pockmarked face was sunken. You could see he was depressed.

In short, the reason that Stalin was so surprised was because he was so committed to both a strategy and an idea of Hitler that were both wrong. His strategy for dealing with the Germans was to enter into an agreement with them, to ensure that they didn't drive eastward. This was precluded by the Nazi's swift westward advance through Europe and Hitler's foolish idea to open a second eastern front (something Stalin didn't think him stupid enough to do). His assessment of Hitler was that of a relatively reasonable man, who he didn't think would subvert the agreements made between the two countries as easily as he did. Remember that even in November and December as Hitler was preparing to formally approve the invasion of the USSR, Stalin and Molotov were still attempting to negotiate with him on the Tripartite Pact, Finnish occupation and Eastern European issues (Romanian oil fields, for example). So when the invasion commenced, even though he had a pretty good idea of it happening at some point in the future, he didn't think it would come either as soon as it did, or in the manner it did. His denial of the invasion, claiming the possibility of a provocation by German generals and not Hitler's order, is evidence of his commitment to this idea and his disbelief at the notion that he was wrong.

Remember also that Stalin was just never wrong. He never admitted wrongdoing and hardly changed his mind on anything. The only times he ever changed his mind on issues were when he was faced with a physical impossibility, and in every instance of this he did so very slowly and grudgingly (and of course never admitting so). This is a great example of that.

Source: Stalin, New Biography of A Dictator. Khlevniuk, Oleg V.