I was recently reading a letter (1832) by Victor Hugo in which he imagines an exchange among the town council citing reasons why an old monument should be destroyed:
"One argues from feudalism, and gets hold of another allegation the tithe; another, the corvée; another, the serfs who beat the water in the ditches to quiet the frogs; a fifth, the right of the first night; a sixth, the eternal priests and the eternal nobles; another, the horrors of St. Bartholomew; another, who is probably a lawyer, the Jesuits, then this, then that; then again that and this; and everything has been said: the tower of Louis d'Outremer is condemned."
So it seems that the St. Bartholomew's day massacre, at least in France, might have been casually brought up the way the Holocaust is today.
Hitler has been famously (if slightly controversially) quoted as mentioning the Armenian Genocide as an act he was willing to emulate on the Eastern Front.
http://en.wikipedia.org/wiki/Obersalzberg_Speech
It was also widely condemned in the west during the genocide itself and for years afterward.
http://en.wikipedia.org/wiki/Press_coverage_during_the_Armenian_Genocide
I wish I had a better answer about whether the Armenian Genocide was as widely known, but it was certainly in the public sphere of knowledge as a horrible atrocity before the Holocaust.
Sorry about using wikipedia as a source, I think that stuff is mainstream enough that it shouldn't be inaccurate.
I think if you're looking for previous examples of "proper" genocide by way of what is now the accepted modern definition, i.e. the deliberate and systematic destruction/extermination of a religious, cultural, political, or national group of people, you'd have to refer back to Biblical examples. "The Ban" was a term used in Hebrew texts to describe the full-scale destruction of an enemy following a clash (see Numbers 21:2). Many biblical scholars have pointed to the aftermath of the Battles of Jericho and Ai as early recorded examples of genocide. Much like what was sometimes seen during Genghis Khan's campaigns in Asia, these examples are all tied to military conquests, and were a byproduct of the same. Some might argue that the Shoah (Holocaust) perpetrated by the Nazi German state in the 1930s and 40s was also tied to military expansion, yet there's ample evidence to show that Hitler's government pursued its genocidal efforts well beyond its army's high-water mark, and that it was an effort wholly separated from expansion and military conquest. So, pre-1945, at least in Western history, you'd have examples like the Siege of Jerusalem in 1099 as maybe the most horrible example of wartime pseudo-genocide. In Asian history, take your pick among your favorite Genghis Khan stories, I suppose (sorry, I'm a bit short on sources there). [Sources: Christopher Browning, 'Ordinary Men'; David Nicolle, 'The First Crusade: 1096-1099'; The Bible]
The tragedy and horror of the Thirty Years War has always lingered in the minds of many Germans, who were the center of much of the conflict of the war. Pillaging, rape, mass casualties of Civilians were common characteristics of the war. It reshaped much of Germany, particularly Brandenburg, and was one of the last fundementally religious wars in Europe. Its dark scars left a deep impression (at least, on Europeans and Germans) that I do feel it was a recounted atrocity among people before the Armenian Genocide and the Holocaust.
As a side note, many Germans in the late 1900's still considered the Thirty Years War a worse tragedy for their country than the Black Death, and both of the World Wars.
Source: Thirty Years War by Peter H. Wilson
While researching the history of Medieval Polish Jewry, I stumbled upon a book published in 1899 about the topic. I don't recall the title of the book, but in the prologue there was a lot of emphasis on the Chmeilniki revolts of the late 1600s, where several hundred thousands of Jews were killed and how every community felt the impact. So at least for pre-WW2 Jewry, they considered the Chmeilniki Revolts the worst atrocity in (relatively) recent memory.
There really wasn't one. People didn't think about genocide the way we do -- they thought about individual massacres, or racial struggles for land, identity and freedom, not about the planed, systematic murder of millions of people.
That said, in Western Europe before the First World War there were a few stock examples, which included the Roman persecutions of the Christians, the Spanish Inquisition, Aztec human sacrifices, the Terror from the French Revolution, the African slave trade and the Rape of the Congo. Leopold II's actions were probably the most similar in scale to the Holocaust, if not in scope or purpose.
Directly before the Second World War, however, a few more became common: the First World War itself was regarded as an atrocity in some circles, the Rape of Belgium during that conflict played a huge roll in turning global opinion against Germany, various actions during the Spanish Civil War were used (Guernica by anti-Francoists, the Martyrs of the Spanish Civil War for anti-Communists) and, for those capable of finding out what was going on, the Soviet atrocities (like the Holodomor).