I'm not sure it's fair to say that the scholarly consensus has changed on this point. While scholarship on Sparta has become a lot more nuanced and careful than it used to be, the adjustment of the idea that Sparta was a society geared for war is perhaps the most radical element of the new view. There are still active scholars - notably Ellen Millender - who reject it entirely. Paul Cartledge himself (now retired) tells me he remains unconvinced that he was wrong about Spartan militarism.

On the face of it, the Classical Greek sources (to leave Plutarch out of consideration for the moment) seem to confirm the idea that Sparta was totally shaped by the needs of military efficiency. In Thucydides, Perikles already stresses the contrast between Spartan discipline and Athenian liberty (2.39). The orator Isokrates once declared that Sparta was not so much a city as an army camp (6.81). Aristotle asserted that "the entire system of the laws is directed toward one part of excellence, the warlike part" (Pol. 1271b). These are just some examples of the way other Greeks tended to view Sparta.

The view that we should look past such claims is primarily championed by Stephen Hodkinson. He has written a huge amount of scholarship on Sparta, attacking numerous elements of the "Spartan mirage" over the decades of his career (Hodkinson is also now retired). The central thesis of his entire body of work is that, regardless of what outsiders might say, Sparta wasn't actually all that different from the other Greek communities of its time. One aspect of that thesis is his contention that Sparta was not unusually militarised or militaristic - or at least no more so than states like Athens.

The key publication is Hodkinson's essay 'Was classical Sparta a military society?', in Hodkinson and Powell (eds.), Sparta & War (2006), 111-162. In this typically thorough survey of the evidence, Hodkinson contextualises and weighs a lot of these claims, conceding that warfare was important to the Spartans (as it was to other Greeks) but noting that it was nowhere near as total and absolute a factor in Spartan life as modern scholars make it out to be. He points out the limited role of warfare in early sources, the limits of military institutions and customs, the evidence for non-warlike activities, and the wide range of values idealised by the Spartans. His conclusion sums up a lot of what the general paradigm of the "new Sparta" is about:

For most citizens, their role as warriors was only a part, albeit an important part, of a wider range of citizen activities. To characterise Sparta as 'a community of professional soldiers' is consequently too narrow. Notwithstanding the professionalism of the Spartan army, a Spartiate was, in the words of Jean Ducat, not so much 'a professional warrior' as 'a professional citizen.' Classical Sparta was far more than simply a 'military society.'

This is the point on which I elaborate in my discussions of Spartan warfare on this sub, and on which Hodkinson and I have both elaborated in published scholarship. We do believe this point of view is gaining ground, but of course it is up against centuries of scholarly tradition and a great deal of intellectual inertia - to say nothing of people who simply aren't convinced upon their own understanding of the evidence. There is perhaps no simple right or wrong here, only competing theories with greater or lesser personal appeal. But there can no longer be any doubt that older scholarship went too far in its characterisation of Sparta, and the result was too simple and singular to be real.

When/why did the scholarly consensus change?

A very rare instance of lightning-fast turnover that happened recently, by both academic and real-world standards. More can definitely be said about the historiography of Sparta if anyone would like to address the matter; for the meantime, this discussion beneath a top-level post goes a little into why u/Iphikrates did not draw from Cartledge in formulating their post above.

Though, if you'll examine said top-level post, one can argue that the Spartans were legitimately better than the other Greek states in the specific field of hoplite combat. Of course, we then get into the thorny question of whether that counts, on the grounds that given the environment, Sparta was the big fish in a rather small pond...