This is a bigger question than you might realise, in that Soviet agriculture was basically never without serious problems. So while serious shortages and famines were especially notable in the early decades other problems persist even into the era of 'mature socialism'.

At the base of it one of the biggest and enduring problems was a lack of incentive. For the majority of Soviet history agricultural products were procured through some time of price fixed by the state. That price was fixed essentially to suit the state's goals (industrialisation, making the turnover from export better, feeding the cities etc. etc.) rather than based on actual demand. If you cannot get a decent price for your work, you're unlikely to work very hard or do more than the absolute minimum.

This isn't just a question of a lack of a free market either - for example during WWI (pre revolution), during the Civil War and even during the more market-based NEP period similar behaviour was apparent. In these cases the issue wasn't just a question of how much the peasants could sell their produce for, but also that they had nothing to buy. Disruptions in distribution networks, other items being in greater demand etc. could result in rural areas having no access to basic goods that they might need (never mind want) so they didnt have a lot of incentive to sell their goods. Bruce Lincoln talks about this in the context of WWI and the Civil War in "Red Victory", he also talks about how grain that wasn't worth selling would sometimes be turned into samogon - a rough grain spirit which could be easier to sell and unlike wheat wouldn't rot if left sitting around too long.

Grain seizures, the 'solution' to the peasantry not selling their grain (or just being perceived to be hording it) often created other problems. If there genuinely was no more grain to give this could result in seed being seized and hence next season's harvest is going to be even worse. The grain seizure campaigns usually also resulted in deportation, imprisonment etc. which also meant that large sections of the peasant population, and in the case of the campaign against kulaks often the more successful ones, were being removed from the countryside.

Add to those who were involuntarily removed from the countryside those who 'voluntarily' fled, or even those who just plain sought out better oppotunities in the cities. In "Magnetic Mountain" Kotkin highlights how peasants were actively recruited in the 1930s to work on new industrial complexes like Magnitogorsk. Peasants would be offered better pay, signing bonuses and so forth and not unsurprisingly this was more attractive than working on a kolkhoz. Even into the later decades, there wasn't a lot of prestige or good paying goods in the rural areas, adding to the brain drain. This dislocation of anyone with skills or ambition created deeper issues in terms of the quality of work being done.

Party-state interference could go beyond price manipulation and direct seizures. Peter Rutland in "The Politics of Economic Stagnation" noted for example that in the Brezhnev era local party bosses regularly intervened and interfered with the work of collective farms in their area, to the point of trying to tell the farmers, managers etc. when and how to sow their fields. This contrasts with industry, where this sort of 'petty tutelage' or 'voluntarism' was actively campaigned against so was not quite as widespread. Part of this was because the agricultural sector didn't have a powerful ministry to counter-balance the local party apparatus' demands. Put another way, picking a fight with the steel mill was generally harder than picking one with the holkhoz manager. There was also an expectation or understanding that this was a key role of the party leader in rural regions - it would have been hard for them to justify their existence, or show how keen they were, otherwise.

There was also a basic political bias. For example there were various solutions or potential reforms that could have further softened the issues in agriculture. These include expanding private plots of land and the 'link system' of labour/land allocation. While these were experimented with or tolerated during the later years, they were never allowed to take root. The simple reason for this seems to have been it would have undermined existing political structures, and the idea of a more prosperous but non-collective peasantry was seen as intrinsically dangerous so the goals of the CPSU.

Almost forgot something more basic - infrastructure. It doesn't really matter how much grain etc. you harvest if it rots in the back of trucks or in heaps at railway depots. Because investment was more heavily directed to industrial projects and cities, rural areas often had poorly developed roads, railways and a lack of transport (e.g. trucks). Referring back to Rutland again, even attempts to remedy this in the Brezhnev era were often failures. For example more grain silos would be built, but all of the infrastructure that needs to go around those to make them worth using often wasn't built - the campaign used silo construction as the indicator of success... so that is what got done.

Investment was partially a problem, but a complicated problem. Early in the Soviet era investment was biased towards other sectors as noted above. However even in the Khrushchev and Brezhnev eras were agricultural investment increased significantly, productivity and outputs remained poor - to the point where Soviet agriculture was some of the worst in the world. This probably ties back to issues like where that investment was directed, fundamental incentives to work and so forth.

I think that gives a pretty reasonable overview. Did you have any further questions or want further resources?