You're correct to wonder why any peasant or worker would want to side with the Whites because for the most part they weren't motivated to join them and this was a problem for Whites. In fact this leads us right to looking at one of the factors which caused the Whites to lose the war.

The "average person" fighting for the Whites was certainly not representative of the average Russian. The vast majority of the White army was made up of officers, not normal soldiers. In A People's Tragedy Orlando Figes cites the fact that of the 3,000 original volunteers for the White Army, only a dozen or so were soldiers! Officers were made to serve at lower ranks than they had previously held, which caused tension and friction. Jealous officers didn't want to be demoted or made to serve under their social lessers. Many ended up lounging at cafes. Even when they did have low ranking soldiers, they were conscripts from the peasantry, and desertion was a problem.

This was representative of the fact that the Whites approached the war with a strictly military mindset, when it was as much a political struggle as a military one. They refused to update their politics to the post-revolutionary world, and that cost them support which they desperately could have used. They cared so little about what people thought about them that the majority of their propaganda was directed at the Allies, which let the Reds dominate the propaganda game inside the country.

This also extended to food and supplies. Peasants loathed selling food to the Whites, which was responded with violence and grain requisitioning (though this was also a feature of the Red Army as well). The White land policy was to push back land reform, and they re-instituted Tsarist police and squires to lord over the peasants. Nationality also played a factor. The Whites were based outside of the Russian heartland (which was held by the Soviets), in places like Ukraine and Siberia, and among the Cossacks, yet their ideology was centered around a centralized Russian nationalism, which didn't appeal to the national minorities.

In the end, ignoring politics and ignoring the aspirations of the peasants, workers and national minorities, were one of the major factors which contributed to the eventual defeat of the White Army. Its leaders were too rooted in the 19th century understanding of warfare and of the Russian Empire and social order. Their inflexibility made it so for the most part pretty much no "average people" joined the Whites voluntarily.

Sources:

Bushkovitch, Paul A Concise History of Russia New York: Cambridge University Press, 2012

Figes, Orlando A People's Tragedy: The Russian Revolution 1891-1924 New York: Penguin Books, 1996