A good starting point for someone with passing familiarity with the Roman Empire might be SPQR: A History of Ancient Rome by Mary Beard. Beard covers many of the topics that those with passing curiosity might find intriguing. Her history covers early Roman history through about 200 CE. While SPQR is a good book, it's definitely an introductory title. I have two main complaints, which I list below:

1. SPQR does not include any history past 200 CE because the author, Mary Beard, claims that the Roman Empire had changed substantially by that date. It is an incomplete account of the Romans. I wish she had extended her history further to include life in the declining Roman Empire. The Roman civilization changed significantly between its founding and its eventual downfall.
2. The first half of the book focuses on Cicero, the legends of the Kings of Rome, and other important Senators and their letters, while avoiding common topics that would interest a reader like, “What was the Roman culture like?” and “How did the Romans decide on democracy?” Instead, Mary Beard often stops her narrative to explain that history is written by the victor, so we shouldn’t jump to conclusions about any one person or event. It becomes rather tiresome to repeatedly read this. I wish she had mentioned that in the prologue and then spent considerable more time explaining Rome as she did in the last half of the book.

The latter portion of the book is quite enjoyable. It focuses on Roman expansion, the bureaucracy of the Empire, and the life of the average urban and rural Roman. I felt informed and it easily kept my interest until the end. Aside from my minor quibbles above, SPQR is the most accessible history I have yet to read.

You might also consider reading The Rise of Rome: The Making of the World's Greatest Empire by Anthony Everitt. It covers much of the same material as Beard's book but in a more narrative style. Everitt has also published biographies of major Roman figures like Cicero, Caesar Augustus, and Emperor Hadrian.

Keep in mind that both of these books try to be histories of the entire Roman history: Kingdom, Republic, and Empire. Both cram a lot into their pages to try and fit it all in one book. Those points aside, I believe these are great starting points.

I would also direct you to the booklist published by the subreddit here. I often start my search for a new book to read by perusing this list.

For the fall of the Roman Empire, I highly recommend The Fall of the Roman Empire: A New History of Rome and the Barbarians by Peter Heather. It is a great book that takes into account the latest historical evidence while at the same time presenting a well-written narrative of the decline of the Roman Empire. Heather is an expert both at Roman and "barbarian" history, having written the authoritative work on the Goths. Also, unlike other historians who've been writing about the fall of the Roman Empire with the view to debunk Gibbon, Heather is unafraid to follow where the evidence leads him.