I know it sounds like the beginning of a bad joke, but is there any real difference, or are they just synonyms for "poor person"
What time period are you interested in?
I can answer it laconically for the Medieval and Early Modern times.
Peasant: Its a catch all term for a rural person, mostl likely (but not always) living off the land as a farmer. Depending on times, places and circumstances, a peasant could have been poor to the point of starvation, reasonably well off or even rich. By the late medieval period it was not unheard of for peasant children to go to college. In many time periods and countries, peasants would be tied to land in some fashion, and/or tied to a specific lord/landowner. Some peasants would own land, and some would be only "loaning" it from someone else,a nd yet some would be slaves/thralls. In some cases peasants would have very little to no personal freedom. In the early medieval period it is not exactly clear to me what is a difference between a slave/thrall working the land, and a a serf-peasant doing the same, save for cultural/traditional distinctions.
Serf- essentially a servant, someone who's freedom was restricted, and who's main job was decided by his senior. This could co-incide with being a peasant, but not necessarily so, since serfs could be doing various jobs. Serfs were also not slaves, in most circumstances they were granted some minor rights and their relationship with the senior was more akin to "permament employment" or legal bondage, than outright ownership. There were several "degrees" of serfdom between being a freeman who simply owes fealty to a lord, and a slave with absolutely no rights. The ammiunt of rights usually coincided with the ammount of land the serf was given to use.
Pauper - literally a poor person. A pauper might or might not been a lower class of serf. Interestingly, completely free men could have been paupers too, lack of obligation towards a lord also meant no protective economic net to fall onto. It was not unheard of for paupers to actually seek serfdom to fend off starvation and extreme poverty. Some paupers could have been runaway slaves or serfs. In fact, in some places and times like for example Polish-Lithuanian Commonwealth, there was a transient state between a free-roaming vagabond pauper who would become a temporary lowest serf during harvest, and then go back to be "unemployed".
Some sources:
Finberg;. The Agrarian History of England and Wales;
J. Serczyk: the Lives of Polish Peasantry;
R. Duval: The Vagabond and the Serf;
I'll take a shot at this from the perspective of the Western European Middle Ages, with the explicit caveat that this is not an answer about Eastern European and Russian forms of 'serfdom', which are an Early Modern phenomena which lasted right up to the end of the 19th century. Hopefully someone else might come along to expand on that particular subject. I'm adding this particular answer, despite there being another top level-reply primarily because, although I wouldn't say the earlier answer is 'wrong' - it's largely a matter of terminology and semantics, so there is not 'right/wrong' binary - I think much of the discussion, particularly of the terms peasant and serf, are confusing, unhelpful and not particularly in tune with the latest scholarly discussion.
Peasant: The word peasant is a modern one, and the concept is a modern category of analysis. It's not found in any medieval sources, and a wide variety of words, including but not limited to rustici, agricultores, laboratores, villani, coloni, pagani and servi, have been translated as 'peasant' by modern historians. While writers of popular books, textbooks and even scholarly works which are not particularly focused on the distinctions between medieval country-dwellers still tend to use the word 'peasant' as a catch-all for all the common people living in the medieval countryside, most specialist works on the rural economy in the medieval period tend to have a more specialist definition, which is largely an economic one. I'll quote at length here from Chris Wickham's definition, which I think is quite a useful one and one which a majority of specialists would probably (more-or-less) accept:
[O]ne can regard the concept [of a peasant] as a strictly economic one. [A] settled cultivator (or, more rarely, pastoralist), cultivating largely for subsistence, who does at least some agricultural work personally, and who controls his or her labour on the land. [...] Peasants could further either be landowners ('small owners') or tenants; they could be part-time - though not full-time - artisans; they could also have tenants or labourers working part f their land, as many cultivators do, especially richer ones. If a rich peasant, however, accumulated so much land that s/he did not have to farm any of it directly, by my definition s/he would stop being a peasant and become a 'medium landowner'.
You should be able to see from this what difficulty I have with the previous definition of 'peasant' offered in this thread: it's just not precise enough, and therefore pretty useless for any sort of analysis.
Serf: Serf is a super-tricky, terribly defined word which can mean many things to many people. I've already mentioned its use in an Early Modern Eastern Europe, but even in the medieval West it's never used consistently. The previous answer gave the most common/generic definition, that of an unfree peasant who was tied to the land, and that's certain the status of some peasants at some times in history (England in the 12th to 14th centuries, for example) but legal unfreedom is much much more complex and varied than this. As a result, most scholars prefer to talk about degrees of unfreedom, which varied from place to place, and era to era, and which never really gained any kind of consistency in rights, restrictions and obligations. One of the biggest problem with the word 'serf' (and derived words like 'serfdom', to describe an institution) is that, like 'peasant', it isn't a word that was used by medieval people themselves. The most common word which is translated as 'serf' is actually servus - the same word which is used for and frequently translated as 'slave'! However, the word 'serf' was initially preferred by historians to distinguish medieval unfreedom from what they saw as the radically different systems of ancient slavery or (more relevantly) modern, race-based slavery (hat-tip to /u/textandtrowel whose ideas these are). Most historians have emphasised the difference as being one of autonomy and control of one's own labour (note that Wickham emphasises this very fact in the quote above). But, in many cases, it's very difficult to see the difference between slaves and unfree peasants: if you are treated as property, bought and sold, beaten and raped and can potentially have your family split up by the whims of your master, does it really make a difference if you get to decide whether you toil in a wheat field or a vineyard? On the other hand, we can often see free peasants being treated just as badly as their unfree neighbours, so the practical implications of unfreedom are quite difficult to discern much of the time. In either case, the word 'serf', beyond signifying unfreedom, is actually pretty useless as a description and an analytical category, IMO.
Pauper: On the whole, this tends to just mean 'poor' in a general sense, and is the only word here that can be found in medieval sources. Pauperes didn't only mean the poor however, but could also mean, in more general terms, either just 'the common people' or 'the powerless' - the term potentes et pauperes is often used to encompass the entirety of the populace. For some reason, I tend to associate the English word with the Early Modern and Victorian periods, but we'll need a specialist if they're going to accurately explain what's happening in terms of connotations.