There are many choices to consider here depending on what kind of things you are looking for and I'll name quite a few books that are both readable and will give you an accurate sense of what modern historians think about the Romans. I also encourage you to think about the empire as an entity that lasted well into the middle ages - I hope you don't mind some of my later recommendations! To get a general sense of Roman history, try The Romans: From Village to Empire (2012). If you are interested in military affairs, I recommend Adrian Goldsworthy's The Complete Roman Army (2003) for, well, a complete manual on what the Roman army was all about. His volumes on the Punic Wars, Julius Caesar, Antony and Cleopatra, and Augustus are all excellent too and are all very readable introductions to the complex world of Roman politics. To get a sense of historians' views on the empire in the fourth and fifth centuries, consider Goldsworthy's The Fall of the West (2009) and Peter Heather's The Fall of the Roman Empire (2005), both of which also includes quite a bit on the preceding period, though I'm sure a classicist can recommend better books covering the period between Augustus and the third century.

Imperial Rome however lived on after 476! The sequel to Heather's book, The Restoration of Rome (2014), is an excellent overview of what happened after Rome 'fell', which is rather important since the Roman Empire lived on in the east and the post-Roman world in the west was never as bad as what past historians, such as Gibbon, told us. Late antiquity really wasn't as cataclysmic as some people thought it was and this was really down to the amazing work done by Peter Brown, particularly in his The World of Late Antiquity (1971), which dealt with cultural history from the third century to the seventh, so I heartily recommend that as well, as it literally transformed our understanding of the late empire. You might be told that Gibbon is a great way to introduce yourself to the history of the late empire - he is definitely not. Give Brown a try instead, he writes very eloquently and his contribution is valuable even now, to the extent that he should be the name on everyone's lips rather than Gibbon if you want to learn about late Rome/Byzantium. If you are interested in Christianity as well, which cannot be separated from any discussion of the Roman Empire from the fourth century onwards, Brown's The Rise of Western Christendom, 200-1000 (2002) is the book to read. Of course, late antiquity was also an age of violence and material deprivation despite its cultural and political continuity with the earlier Roman period. This is best encapsulated in Bryan Ward-Perkins' The Fall of Rome and the End of Civilization (2006).

For the Byzantine Empire (though it really was just the Eastern Roman Empire, historians just use it to distinguish between two phases of the same empire), consider Norwich's A Short History of Byzantium (1997). Even though it is a bit too old for my taste and only provides a very basic narrative through a surface reading of the sources, it will at least introduce you to the key events of the empire's history. The empire lasted for more than a thousand years, so it is obviously quite difficult to summarise them in a readable way - Norwich in my opinion did the best job he could in the circumstances. For a more modern approach, I heartily recommend Judith Herrin's Byzantium: The Surprising Life of a Medieval Empire (2007), as she examined the empire's history thematically rather than chronologically, and looked at fields often unexplored in popular histories of the Roman world, such as on the role of women, art and the experiences of the average citizen. Averil Cameron (one of the biggest names in Byzantine studies) also wrote The Byzantines (2009) and it is very good as well. It is supposed to be an introductory text aimed at both non-specialists and Byzantinists, but I think Herrin's book is more readable and provides a more innovative approach to studying the empire. Still, give them both a try and hopefully one of these books will get you enthused about everything Byzantine! For more specific treatments of certain periods, Mark Whittow's The Making of Orthodox Byzantium, 600-1025 (1996) and Peter Frankopan's The First Crusade: the Call from the East (2012) are both very accessible and good overviews of the specific periods they cover.

Ancient Rome: The Rise and Fall of an Empire is the best book I've read for this.

It goes from the tiny rural beginnings all the way through to the fall of Rome, focusing on the key turning points in history all the way through. It really helped me get a skeleton framework in my head of the main story of Rome, that I was then able to hang more details on as I read further into each time period.

Have you checked out our [book list] (http://www.reddit.com/r/AskHistorians/wiki/books/europe#wiki_ancient_rome)? There are a few texts listed that provide great introductions to Roman history. If you are looking for more specific recomendations, are there any particular topics or time periods you are interested in?

I've got Livy: The Early History of Rome, and Tacitus: The Annals of Imperial Rome. Both published by Penguin. Keep in mind that these are primary sources written by people who lived during the Roman period, and they could be difficult to read if you're not used to older style English. These don't read like textbooks, but they make undergrad Poli Sci student's read them; so it's far from impossible.