This is really tough to answer as an AH question, not least because it requires some kind of objective ranking of subjective things. The main thing I'd like to point out is that taking such a class-centric approach to "who really had it bad?" ignores the fact that there are other lenses in play. I'm going to be annoying and quote my answer to an earlier question, "In the Tudor times in England, would a male servant to a wealthy household have any sort of "male privilege" over the patriarch of the family's wife, or would she be higher and better regarded than him on all levels despite him being a man and her being a woman?"

There's a basic problem with the premise of this question because the standard you seem to be using for the concept of male privilege is badly flawed. Male privilege isn't just something men have "over" women - it's the flipside of women's oppression, the lack of disadvantage that is invisible to the people who have it, so that it can be taken for granted. It's easy for someone who possesses male privilege to see it when it's presented as something in action against a woman: a man putting a woman down, a man being believed in a claim against a woman's word, a man being allowed to do something to a woman. It's much more difficult to conceptualize as a "freedom from". While people typically possess some privileges and lack some others, that puts them into a complex web rather than a simple hierarchy of "more important" vs. "less important" oppressions; the latter seems to be implied by this question, as though the servant's male privilege is canceled out by his class standing.

So, one very major way in which this male servant has privilege denied to his employer's wife is that he has control in his sexual relationships. In the Tudor period, the legal understanding of rape was largely constructed to serve the needs of aristocratic patriarchs: the purpose was to punish men who deflowered or eloped with their unwilling and even willing daughters and therefore lessened their value on the marriage market, or who assaulted or consensually slept with their wives and therefore inflicted damage to their own honor. Rape cases rarely went to court - under Henry VIII, they made up about 2.5% of court cases - and accused rapists generally went free. Meanwhile, marital rape was not recognized as a crime because the wife owed her husband her body; women who sought legal separation from their husbands had to prove extreme mistreatment above and beyond non-consensual sex. What does this mean for our characters? The male servant has sexual autonomy. He can sleep with female servants or sex workers without risking his reputation; if he is accused of rape he will almost certainly not be punished, and it might simply be chalked up to fornication instead, solved with some penance and marriage. On the other hand, the lady has little right to her own body. Her husband may force her to have sex against her will, and there is no recourse for her. If she runs away with a lover or is abducted by an enemy, the result is the same: he has the right to bring her back and prosecute the person she left with. (A related issue: married women could not write legal wills, because their husbands already owned all of their property, down to the last trinket. Even a poor man could leave his belongings as he wanted when he died.)

More abstractly, this male servant is free from being told that he is physically, mentally, and spiritually inferior because of his gender. Women of all stations were bombarded with this knowledge from a very young age - that God had decreed that they were to be submissive to husbands, fathers, and other male relations; that they were physically and morally weak, easily overcome - to such an extent that their surviving letters frequently refer to it. Now, this is complex. Many women certainly knew themselves to be capable of anything, many women were likely using these tropes in their letters to win sympathy, and many women acted out of their own agency in running great estates and managing large households without masculine pushback. But they were still enveloped in a misogynistic culture that reminded them at every turn of how incapable and inferior they were. To not be subjected to that is a privilege, even if it's balanced out with cultural narratives of inferiority by birth.

Heck, the fact that this hypothetical male servant will never become pregnant and risk his life in childbirth is an aspect of the male privilege that he has, even as a servant!

My point being - you can't just draw a line between the rich and the poor, because there is no one rich experience (or, for that matter, one poor experience). You can also see problems and difficulties in the lives of individual or groups of rich people without concluding that historical rich people as a whole need to be cut slack.