There are many choices to consider here depending on what kind of things you are looking for and I'll name quite a few books here that are both readable and will give you an accurate sense of what modern historians think about the Roman Empire. I also encourage you to think about the empire as an entity that lasted well into the middle ages - I hope you don't mind some of my later recommendations!

To get a general sense of the empire, try The Romans: From Village to Empire (2012). If you are interested in military affairs, I recommend Adrian Goldsworthy's The Complete Roman Army (2003) for, well, a complete manual on what the Roman army was all about. His volumes on the Punic Wars, Julius Caesar, Antony and Cleopatra, and Augustus are all excellent too and are all very readable introductions to the complex world of Roman politics. I have to caution you against reading Tom Holland's Rubicon (2003) here, as despite its popularity it was not written by an academic historian nor written in a way that meshes well with modern academics' conclusions. To get a sense of historians' views on the empire in the fourth and fifth centuries, consider Goldsworthy's The Fall of the West (2009) and Peter Heather's The Fall of the Roman Empire (2005).

Imperial Rome however lived on after 476! The sequel to Heather's book, The Restoration of Rome (2014), is an excellent overview of what happened after Rome 'fell', which is rather important since the Roman Empire lived on in the east and the post-Roman world in the west was never as bad as what past historians, such as Gibbon, told us. Late antiquity really wasn't as cataclysmic as some people thought it was and this was really down to the amazing work done by Peter Brown, particularly in his The World of Late Antiquity (1971), which dealt with cultural history from the third century to the seventh, so I heartily recommend that as well - it literally transformed our understanding of the Late Roman Empire and is very well-written. If you are interested in Christianity as well, which cannot be separated from any discussion of the Roman Empire from the fourth century onwards, Brown's The Rise of Western Christendom, 200-1000 (2002) is the book to read. Of course, late antiquity was also an age of violence and material deprivation despite its cultural and political continuity with the earlier Roman period. This is best encapsulated in Bryan Ward-Perkins' The Fall of Rome and the End of Civilization (2006).

For the Byzantine Empire (though it really was just the Eastern Roman Empire, historians just use it to distinguish between two phases of the same empire), consider Treadgold's A History of Byzantine State and Society (1997) and Norwich's A Short History of Byzantium (1997), though both are a bit too old for my taste. For a more modern approach, I heartily recommend Judith Herrin's Byzantium: The Surprising Life of a Medieval Empire (2007), as she examined the empire through various themes rather than chronologically, and explored fields often unexplored in popular histories of the Roman world, such as on the role of women, art and the average citizen. For more specific treatments of certain periods, Mark Whittow's The Making of Orthodox Byzantium, 600-1025 (1996) and Peter Frankopan's The First Crusade: the Call from the East (2012) are both very accessible and good overviews of the specific periods they cover.

I'll post the same answer I did last time:

The /r/askhistorians reading list is decent. The Mackay book is garbage though, in my opinion. There are several suggestions on where to get started though.

My main suggestion is to find a broad overview book; something like the Oxford History of Rome is not bad. There are also lots of more general histories of the ancient world that contain decent intros, and the great news is that these types of histories, from Oxford, Cambridge, and Harvard Presses, are usually available for a song in used bookstores. Just keep your eye open and pick one up. Once you find a period you really like, then you can start moving into more detailed stuff. Generally the big areas are Republican Rome, Civil Wars, Imperial Rome, and Fall of Rome (West). Most of the books and biographies about Rome fall in this area. There are also other types that focus on culture, or my favourite, architecture. Alternatively, another suggestion is Mike Duncan's 'History of Rome' podcast, which is a great idea if you are someone with long commutes. The first 30 or so episodes have pretty mediocre writing quality but he really picks it up near there. If it was written out it'd be a pretty good and balanced overview of Roman history. My main area of interest is in the Republican era, and then the fall of Rome. I don't find the civil wars that appealing, and most of the Imperial era, while interesting when I was a child (emperors are always interesting when you are young) is not as apealing to me as I got older. For the early Republican Period, I really recommend Klaus Bringmann's 'A History of the Roman Republic'. For the Civil Wars, Holland's 'Rubicon' or Goldsworthy's 'Caesar' are both good and well liked. Imperial, I can't even guess who is modern right now, as I haven't really read a book on Imperial Rome directly in years. For Late empire, you are looking at Goldsworthy and Peter Heather. Peter Heather in particular is very easy to read, and I recommend his books to anyone interested in history. I hope that helped.