The answers I see below attempt to address whether or not life in the Middle Ages was "dirtier" and "mostly unpleasant" compared to the Roman Empire -- but they don't address WHY the Middle Ages are considered that way, which I think is your actual question -- and I think that's a much more interesting question anyway. How we conceive of past historical eras tells us a lot about our own culture(s) -- and you're right, today, in 2014, despite plenty of evidence to the contrary, we do have a popular conception that the Middle Ages were a dirty, miserable time period, especially compared to the "glory days" of the Roman Empire. So why do we think that?

We can trace the roots of the idea back to the mid-1300s, when Petrarch (1304-1374), an Italian poet and one of the earliest humanists, was writing. He criticised the literature of the post-Roman world, and developed the concept of a post-Roman "Dark Ages" when, in his opinion, the light had gone out of literature -- with his hope that his own period, the beginning of the Italian Renaissance, as a return to the brilliance of the Roman world.** Petrarch's thinking was highly influential on contemporary and later writers of the Italian Renaissance (and beyond). One of these writers, Leonardo Bruni (1370-1444), wrote one of the first "modern" histories, The History of the Florentine People, where he divided time into the Roman period, the period of tenebrae or darkness afterwards and now, he argued, the new era of the Italian Renaissance. This 'tripartite' division of history has stuck with us -- classical, middle ages, modern -- ever since, although looking at the dates of the two men I just mentioned, you can see that we've been pulling that middle period forward in time.

The degradation of the "Dark Ages" in the public mind, and particularly in the English-speaking, Protestant mind, continued after the Reformation, when many Protestant scholars and thinkers sought to portray the period after the fall of Rome as a time of unbridled control by the Catholic Church, when the avarice and immorality of popes held back learning.*** For example, the Madgeburg Centuries, published by Lutheran scholars from 1559-1574, identified the pope as the Anti-Christ and also told the legend of Pope Joan as a way to argue against the decadence of the Catholic Church.

In the 18th century, Enlightenment thinkers such as Gibbon in his History of the Decline and Fall of the Roman Empire (1776-1789) and Voltaire in his history, An Essay on the Manners and Spirit of Nations (1756), also spoke of the 'Dark Ages' or 'Middle Ages' as a time when the corruption of the church had prevented any social progress. Voltaire wrote that, when the Church held sway,

there existed great ignorance and wretchedness--these were the Dark Ages

In the later 18th and 19th centuries, particularly in Victorian England, there was a revival in 'medievalism'****, which looked back on the 'Middle' or 'Dark' ages as a romantic time. Although it is outside the scope of this answer, if you're interested in reading about it, I recommend checking out the work of Joanne Parker on the Victorians' revival of the story of Alfred the Great. Scholarship from the late 19th centuries and then through the 20th century up until today has shifted to focus on how the Middle Ages were not so dark after all; hopefully non-academic popular culture will soon catch up to this.

Relevantly, the British Museum has just revamped their exhibit on the period, for example.

*I was going to cite some sources here but realised that the idea of the "Dark Ages" has been so dead in historiography for so long that I couldn't even think of a modern source. Instead, I'll give you this quote from a 1942 paper in Speculum by Mommsen, 'Petrarch's Conception of the "Dark Ages"':

It is important to note, however, that in the latest (the fourteenth) edition of the Encyclopaedia Britannica the term ‘Dark Ages’ is no longer used. On the contrary, it is explicitly stated that ‘the contrast, once so fashionable, between the ages of darkness and the ages of light has no more truth in it than have the idealistic fancies which underlie attempts at mediaeval revivalism.

** Petrarch did not have a high opinion of the post-Roman world at all, in fact -- one thing he wrote about was his ascent of a mountain in the Alps and he claimed to be the first person to enjoy climbing a mountain for the sake of climbing a mountain because no one before him could have understood the emotions involved. This has always struck me as being a supremely arrogant statement.

*** You can hear echoes of this mentality in some of the things that /r/atheism has to say, as /r/badhistory often deals with (but that's a tangent I won't go on here).

**** You can see this in Victorian architecture -- take the the British Houses of Parliament, built in the mid-1800s, which are strongly neo-Gothic in style.

Well, for the most part people were dirtier. The Romans were a very fastidious people about bathing (we have a ton of evidence for this for the elite, but it spreads to others parts of societies as well). Many elite Roman houses would have had small baths, and every city had a bathing complex as part of its urban fabric. Baths were up there with temples, fora, and walls as the big markers of urban space.

However, these baths broke down at a certain point. They were difficult to maintain, especially when they were public baths that relied on the generosity of a civic elite to maintain their functioning. Once this elite stopped endowing/repairing buildings, which happens in late antiquity (no new non-Church buildings, on the whole, after c. 300, although there are always exceptions). In some areas, like the soon-to-be Islamic East, these massive public baths were replaced by smaller private baths that seem to have been set up for profit--you would either pay to go or have a membership. The mechanisms are unclear but the physical evidence is laid out quite nicely in an old but still relevant article by Hugh Kennedy, "From Polis to Madina" (1985).

So the disappearance of the baths and public bathing probably leads to most people being a little dirtier, and as for them being unpleasant...well, assuming you mean the people themselves and not their lives, I'm not sure what this is supposed to mean. You could say that a knight is a lot less pleasant than a similarly upper class Roman because, well, the Roman can read and recite Virgil to you and discuss literature (as long as it's Virgil, or Cicero), etc., and the knight is probably illiterate and superstitious. A Roman Senator or 'knight'/equites (not the same as a medieval knight) will definitely be more cultured. But you would probably have way more fun hanging out with a medieval knight. Upper class Romans were somewhat...disdainful. Especially of people who couldn't speak Latin. So you might be SoL.

It's also pretty important to realize that when people talk about the educated Romans in their fancy houses with togas, that's like .001% of the population. Most people are peasants. 80%+ live outside cities, even at the peak of ancient urban life. Their lives are probably not too different from medieval peasants.

If you're discussing the LIVES of people, then I would argue (many would disagree) that the middle ages were much more unpleasant. As terrible as Roman society was in many ways, it did create a massive amount of stuff. The Monte Testaccio in Rome is comprised of the remains of 57 million Spanish jars carrying wine/oil to the capital. Everyone was eating off fancy plates and lived in houses with roof tiles. Bryan Ward Perkins, in "The Fall of Rome and the End of Civilization," does a very good job of showing how destructive the collapse of the Empire was to the standard of living in Western Europe. Basically, when it came to non-luxury goods, any Roman was better off than an Anglo Saxon King (the king still had some gold though, y'know?).

Some have argued (or suggested) that the end of Roman taxation made life much more pleasant for Roman peasants, who now only had to pay rent to their feudal lords and not taxes to the state above that. Chris Wickham hinted at this in his early, more directly Marxist, work. I disagree, and anyway by the high middle ages I'm pretty sure that tax collecting was on the rise again. But that's not my area of expertise.

Narrative hasn't been mentioned yet here and I think it plays a huge role. Popular history, and especially science history, often follows a whig narrative. We see this especially with the idea of the Norman Yoke--a vision of a progressive "common law" Anglo Saxon England much like the cozy shire from the Lord of The Rings, invaded by the evil Normans and subjugated to backwardness and feudal misery.

Indeed the Norman Yoke was put to great political use by Edmund Burke in the 18th century, and many American revolutionaries.

The basic narrative is that there was a golden age in history, the classical era, where people were scientific and rational. Then the dark ages came where people were backward, dogmatic and stupid. Then the enlightenment happened, The United States was founded, and history ended with the landing of the man on the moon.

Carl Sagan's Cosmos for example follows the whig narrative to a T.