If you look at the data about the income gap between rich and poor in the West and the rise of the social state in Europe, you'll see that it corresponds in time with the boom of the Soviet union and the other socialist countries.
To what extent was that coincidental and to what extent was it promoted consciously by the western governments to lessen the appeal of socialist ideas?
Conversely the widening of the income gap and the collapse of the social state in Europe and elsewhere coincides with the collapse of the socialist countries. Is this unrelated?
As a communist myself I've always heard by fellow communists that the social state has so particularly flourished in the Scandinavian countries because of the proximity to the USSR. Is there any grain of truth in this?
Yes, to some extent. There was a conscious effort to come to a compromise whereby workers rights were protected and the capitalist class as well. This was done by centrist parties aiming to undercut the appeal of the far-left. This happened in the US as well, by the way. It's not really that people were looking to the soviet union as a model, but that there were domestic labor movements that had started before the Soviet Union had come about. In fact, many communists bemoan these policies for the exact reason that it reduces the appeal of communism. For example Marx wrote:
"However, the democratic petty bourgeois want better wages and security for the workers, and hope to achieve this by an extension of state employment and by welfare measures; in short, they hope to bribe the workers with a more or less disguised form of alms and to break their revolutionary strength by temporarily rendering their situation tolerable." -1
So basically it's politics, but isn't that what democracy is really about anyway? Each party tries to get votes however it can and they all try to get what their constituency wants, coming to some compromise. And besides, it wasn't all politics, there were other reasons to support the welfare state that included Christian social policies which basically supported health care and food for the poor, fuller employment, etc. We see this continuing today in Europe in the form of Christian democratic parties like Merkel's CDU.
The collapse of the social state movement in the US probably pre-dates the collapse of the Soviet Union in my opinion, but that's getting into the 20 year rule for this subreddit.
1: https://www.marxists.org/archive/marx/works/1847/communist-league/1850-ad1.htm
I hope someone else can go into more detail, because I find this a very interesting topic of discussion and I know I cannot research it adequately.
I think it goes back further, at least to the social reforms of the latter third of the 19th century, perhaps further to the erosion of magnate wealth. But the two world wars had a huge impact through taxes and high employment during and after the second.
I'd say there are three elements: an attempt to sustain the political status quo by stealing some of the clothes of the left and thereby de-clawing it to some extent; the political and industrial action of socialist and labour movements themselves (as well as sincere bourgeois reformers); and recognition on the part of some economists and policy-makers that extreme inequality was no longer good business sense even if it had once served to concentrate capital for investment. Wartime national solidarity and sacrifice too had an impact, notably in Britain, even traditional Conservatives embracing the conviction that there must never be a return to the 1930s.
There is also an argument that demography played a key role, with falling mortality "squeezing" the working-age population and increasing their bargaining power, industrially and politically, until the post-WW2 baby boom from the late 1940s to the early 1960s reversed the pattern as its offspring entered the labour market in later decades. If so, we may face another reversal as those past working age - the very same boomers - increase, though the availability of vast globalised labour forces elsewhere may mean differentials stay as they are or continue to widen.
The later divergence seems to precede the Soviet collapse: in the UK its onset is largely associated with Thatcher's premiership, in the US it's been dated to 1968. But the subsequent collapse of the traditional left has certainly weakened the response to it (though the western left's troubles too pre-dated 1989 as the Cold War widened divisions within it, splitting social democratic parties, union confederations and later Communist parties themselves).
Scandinavia is indeed interesting. There I think the reasons are largely historical and cultural. While there were great medieval magnates, peasant subjection never reached the level characteristic of the period elsewhere in Europe, allowing the early development (puncuated by Denmark's inglorious but mercifully brief 18th-century experiment in statutory neo-villeinage) of substantial classes of farmers and free workers whose interests coincided (after much haggling) in the first steps toward social provision.