I am not any kind of Neolithic expert, but I do know that Andrea Matranga addresses this as part of his upcoming paper on the Neolithic revolution (paper here). The invention of agriculture is, in this model, less about learning how to domesticate crops, and more about sedentarism and storage. Staying in one place and hoarding food allows for consumption smoothing across seasons and between years. Hunter-gatherers experienced higher variance and seasonality in food consumption. They ate better on average, which is why they are taller than farmers, but experienced acute starvation more often. This is why we see "Harris lines" more frequently in the bones of hunter-gatherers, which are telltale signs in bones that reflect the stopping and restarting of growth during acute starvation episodes.

It is the point about acute seasonal starvation that is missing from Jared Diamond's "Worst mistake in the history of the human race" narrative. It is good to be tall, and to eat well in the good times, work relatively little, and escape endemic disease. But it is quite plausibly better not to starve nearly to death most years during the off season.