This is something that's still debated hotly by professional historians. No conclusive answer is likely to emerge at least before the Kremlin opens its archives.

Furthermore, there isn't really one but two questions. The situation in 1939 was one thing; but there is considerable debate about whether the war and occupation could both have been prevented if Finland had followed a different policy from earlier in the 1930s or 1920s.

For 1939 scenario, there are essentially five pieces of circumstantial evidence pointing to the conclusion that territorial concessions would probably have been followed by a takeover.

1 - The Molotov-Ribbentrop pact and its secret protocols. These divided the Eastern Europe to German and Soviet spheres of influence and gave the Soviet Union free hand with the Baltic states and Finland. Germans understood perfectly well that this would mean occupation and annexation of the said countries. Germany honoured the pact scrupulously until late 1940, and Soviet Union also did what the secret protocols implied it would do - took over the Baltic states and tried to take over Finland.

2 - The example of the Baltic states. Estonia, Latvia and Lithuania were presented with similar ultimatums as were presented to Finland. These countries acquiesced, let Soviet troops in, and in 1940, were forced to hold "elections" where they unsurprisingly decided to join the Soviet Union.

3 - Soviet leadership and press were known to consider Finland and Baltic states to belong to Russia, as they had been parts of the Russian empire until the end of the First World War. Stalin in particular had been saying that letting Finland go had been a mistake.

4 - The territorial demands put to Finland would have neutered the main defensive line against an invasion from the East, put a strong Soviet base within artillery range of capital Helsinki, and greatly complicated (if not made impossible altogether) any hope of foreign help. Furthermore, the Soviet press and leadership had been campaigning against Finnish government and the Finnish "German sympathies" and "Russophobia" (which were true enough: most Finns were very deeply suspicious of the Soviet Union, which was feared as the heir to Czarist imperialism AND as the seat of communism). To me at least, it seems likely that these would have been provided a nice pretext for additional demands, just as they did in the Baltic states.

5 - During 1940, after the Winter War, the Soviet pressure continued heavily, culminating in scuppering an attempt of a defensive treaty with the Swedes, attempts to influence presidential elections, a shootdown of a Finnish airliner, and in troop concentrations to the border that were very nearly met with a general mobilisation. These, combined with German occupation of Denmark and Norway (which isolated Finland even further from the Western allies) and fall of pro-Finland France, were very important factors in driving Finland to ally with Germany, which by late 1940 was already preparing for Barbarossa and welcomed a free ally. Reportedly, Soviet foreign minister Molotov asked the Germans for a free hand in Finland in 1940; this time it was refused. Many contemporary and later observers are certain that without German support, the Soviet Union would have occupied Finland in 1940 or early 1941.

Now, the "Finnish policy changes earlier" scenario is murkier and people are prone to interpret it so as to suit their preferences. It is clear that the Finnish government did not understand the gravity of unofficial Soviet feelers put forward in 1938. In these, the Soviet Union expressed its concern that in the event of general war, Germany would invade Finland in order to conduct operations against the Soviet Union and Leningrad in particular. The Soviets wanted "guarantees" that Finland would not assist Germany in a war against the Soviet Union, and would actually resist a German invasion. In this case, the Soviet Union would provide economic and military assistance and pledge to remove its troops once the war was over.

However, these overtures were made by a Soviet diplomat Boris Yartsev, who was officially a minor functionary in the Soviet Legation in Helsinki. In reality, he was a member of the NKVD and kind of a special envoy; but the Finnish goverment didn't understand this and did not take him seriously. Yartsev received a boilerplate answer that Finland would honour its neutrality policy and reject any mutual assistance treaties, but could provide a written assurance it would not permit any great power to use its territory. Both historical assessments and archival findings indicate that the Soviets did not believe Finland could do so, and doubted whether the Finns would even want to resist a German invasion.

From Soviet perspective, Finland had almost been taken over by a far right coup in 1932 and had outlawed and suppressed communists, while the press and public opinion were in general hostile to the Soviet Union and at least somewhat pro-German. There was no understanding whatsoever for the Soviet demands among the general public, and only a poor one in the upper reaches of the government: notable exception was Field Marshal Mannerheim, who as a Czarist general understood perfectly well the problems of defending Leningrad. He was in fact the most vocal proponent for acquiescing to Soviet territorial demands, offering to explain in public that he'd take full personal responsibility for the decision.

It can be argued - and it is argued - that if Finland and Finnish government had been more active in building better relations with the Soviet Union starting from the 1920s, the distrust might not have been as great and territorial concessions might have sufficed. It is however hard to see how this could have been achieved domestically, given the memories of the Finnish Civil War and the resultant suspicion of communism in all of its forms. In addition, Soviet Union was headed by a paranoid dictator who very well might have pressed for further demands anyway.

Molotov himself recalled later that a) Finnish willingness to fight came as a very big surprise to the Soviet leadership who genuinely expected few symbolic skirmishes followed by a parade march to Helsinki, and b) it was this willingness to fight (and the skill with which the fighting was conducted) that made him to conclude that occupation of Finland would be an open wound for the Soviet Union, to be avoided if possible.

Sources

There are many and this is a synthesis of more than a dozen books I've read, but a good and modern English synopsis is

Vehviläinen, O. (2002) Finland in the Second World War: Between Germany and Russia. (Transl. Gerard McAlester). Palgrave.