It seems the majority of these type of events that are so commonly known only ever come from European history. I'm most interested in those that occurred in other regions, especially those that truly took a toll on specific empires.
The "First Cholera Pandemic" of 1817-24 may have killed up to 100,000 people. It began in Calcutta and with the help of British troop movements and the travels of religious pilgrims, it spread to Japan, Syria, Nepal, Afghanistan, China, etc.
Then there was the 2nd Cholera Pandemic of 1829-1851 which probably killed as many as the 1st.
And the Third Cholera Pandemic (1852-1860), which killed over a million people in Russia alone. This is the famous one where a doctor (John Snow) saw a lot of cases of cholera around a well and had the pump handle removed, thus leading to a decrease in cases and the start of epidemiology.
There have been a number of yellow fever epidemics in the U.S. Even here where I live in Norfolk, VA in 1855. There's a lot of information on that one in particular here: http://www.usgwarchives.net/va/yellow-fever/yftoc.html
These don't amount to the numbers taken by the black plague but it's a start!
The Roman Empire experienced a large plague during the reign of Marcus Aurelius. It is highlighted in the Historia Augusta and a couple other sources I can't remember at the moment. Supposedly the plague was carried by troops coming back from the east and it decimated the economic system contributing to instability during the reign of Commodus.