Alright, this question is up for a while without a response, so I figured it’s time I give it an answer. A few qualifiers:
This isn’t exactly my exact area of expertise, so while I have a lot of insight into it, I’m not familiar with the cutting edge debates in the literature on this topic, more a sort of textbook knowledge of the topic with some useful sources I can at least recommend for further reading
It is very hard to do this question justice while respecting the 20 year rule. I’ll do my best, but much of this transition is still unfolding, so the truth is we’ll only have enough distance/access to the needed archival documents (assuming Israel gets better at declassification) in many many years.
I’m referring to the early Israeli ruling parties (such as mapai and Mapam, or Labor which was the biggest faction in these parties) as left wing because they were socialist, and that’s how they referred to themselves. However, it’s very possible to argue on issues of security/military etc. to argue these parties weren’t very left wing at all, so I want to make sure I acknowledge that.
So with that out of the way, the two major historical forces which lead to the near irrelevance of the dominance of Israeli socialist governments were the lefts inability to appeal to the changing demographic groups in Israel, and the perceived failure of the left’s approach to security. The security question is far more straight forward, and ultimately less interesting (and I believe less significant, but that remains up for debate) so I’ll mainly be focusing on demographics. However, the one sentence version of security: the left is remembered for major security failures in Israel, specifically the failure to anticipate the surprise attack of the Yom Kippur War, the failure to quickly put down the intifada, and the failure of the Oslo accords to bring peace. All of these “failures” can be questioned: how much was it truly the left wing governments fault, would it have happened to any government, were these “failures” inevitable—however, for many Israelis the blame falls on the left wing parties lap.
Now, onto demography. To be clear, demographics are not destiny. There really is no reason the socialist parties couldn’t appeal beyond their shrinking base of upper class Ashkenazi support, and to the 4 ethnic groups (Mizrahi Jews, Russian Jews, Ultra Orthodox, and Arab non-Jews) which have grown significantly in size since the founding of Israel in 1948, but for a variety of reasons they have failed to, and today the first three of these groups vote overwhelmingly for right wing parties, and the third group tends to not vote, or vote for parties which don’t sit in governmental coalitions.
The most significant of these groups is Mizrahi Jews, or Jews from Middle Eastern and North African countries. These Jews mostly migrated to Israel in the first years of statehood. While as professor Devin Naar eloquently states a recent interview with Jewish Currents there was nothing inherently right wing about this group of Jews, the way events unfolded would push much of this demographic into the right wing camp. Mizrahi Jews felt overlooked by the Ashkenazi elite upon moving to Israel. Many had to live in transit camps for a number of years, and perceived the lack of aid as a product of the indifference of the ruling labor party. This wasn’t entirely without justification. David Ben Gurion famously quiped that there was nothing wrong with living in a tent for a while when discussing the conditions in the transit camps. This glibness was in contrast to conditions which were described by a Haaretz reporter who (in a much discussed report at the time) snuck into a camp and described the conditions as truly horrible.
While this wasn’t enough to immediately turn Mizrahi Jews against Labor (who had a fairly effective political machine to recruit votes) the continued poor conditions of Mizrahim, along with outreach from the Herut party, eventually pushed them into the right wing camp. In Amos Oz’s book In the Land Of Israel he records interviews with Mizrahim who voted for Israel’s first right win minister, Menachim Begin, in 1977. They discussed the disrespect they continued to feel from the Ashekenazi left, and for many their vote seemed to be as much against Labor as it was for Begin. However they also contrasted this disrespect with the sympathy they heard from Begin, who acknowledged their struggle, and acknowledged, and shared their religious identity as opposed to the strongly secular left.
A slower developing problem for the left was the growing Ultra-Orthodox and Orthodox population In Israel. Early on in Israel’s history (and again briefly with the formation of the Shas party in 1984) these parties did side with some left wing governments, but there was a much more natural alignment with the mainstream right wing parties like Herut/Likud, which had always been more religious, and less supportive of issues of women’s rights secular marriage (and later gay rights) which are mostly opposed by the political leaders of these communities. A lot of commentators focus on the large families many members of these communities have, which is certainly a factor in their growth, but immigration (notably from the US as discussed in Sara Yael Hirschorn’s book City on a Hilltop) would lead to not only a significantly larger Orthodox population but a more politically radical/right wing one, as American orthodox Jews became some of the most hawkish individuals in the orthodox/Ultra Orthodx population. The alliance of ultra-orthodox parties with the right wing in Israel give the right wing parties a “head start” in coalition building. Since many ultra orthodox voters vote for these parties in every election it guarantees a significant number of seats for a right wing coalition, and narrows the number of seats a right wing party needs to get the 61 seat majority in the Knesset.
Next with the fall of the Soviet Union nearly a million Jews from the former USSR moved to Israel. While I’m having to steamroll over complexity of all these demographic groups, Jews from the USSR were perhaps more than any of these other groups extremely likely to vote for right wing parties regardless of Labor’s outreach/policies. Although Russian Jews were largely supportive of secular policies (and hence would go on to form their own right wing staunchly secular party) they were just extremely unlikely after their own experience in the USSR to vote for a socialist party. Much like Russians who moved to the US and supported Reagan, these Jews from the USSR gravitated away from left wing parties and towards the right wing. There are also those who argue that their political worldview shaped from the USSR meant they were more comfortable with the idea of a “strong leader” and less concerned about democratic institutions (which aligned with the right wing in Israel), but I’m less sure about this argument
Finally, the left has never made a good faith effort to include much of the Arab population in their governing coalitions. Some Arab parties have supported the left wing coactions from outside the government under Rabin, but the political taboo of including an Arab party in a coalition has never been broken in Israel. To be fair Arab parties aren’t exactly jumping at the idea of joining a governing coalition, but the Israeli left has given them little reason to want to. Despite massively growing as a percentage of the population, and being slightly more sympathetic toward the left wing parties in Israel, the Israeli left has done little to reach out or include Arab parties. So while the right has its bedfellow of ultraorthodox parties, the left has no equivalent.
There’s a LOT more I can’t cover here, not just security, but changing ideas of capitalism, lack of leadership in the left, the diminishing importance of the Kibbutz in Israeli society, but ultimately these other issues are just more examples of how the socialist parties have failed to appeal beyond a shrinking slice of the Israeli demographic pie, and why they have found themselves increasingly irrelevant in Israeli politics.