When you read a lot about your expert topic you are likely to have read so much about the subject that you will find it easy to connect the dots. However, often when I read books on parts of history I am less familiar with, I find it hard to remember things after a while, even though my memory is surely not bad. I would be interested in how you manage taking/organizing your notes on what you read (assuming you do take notes). Do you write annotations right into the book? Do you write short notes on pieces of paper and store them in the book? Or do you write summaries later on on the computer? Do you use a personal wiki software or something like that?
I use Zotero, which basically lets you import the books bibliographic data and then keep notes attached to those books. It's pretty good software. Endnote does the same thing, I believe.
When I'm starting on a particular time period or event that I'm not as familiar with, I usually make a running timeline of events and a general list of people (which will usually include birth/death dates, their titles, and any other notes that are relevant). Since I am easily distracted, I generally don't use the computer while working and generally do this on paper. I may also keep a running list of other books/articles cited in this book that I want to check out. If I own the book, I'll also make some notes on post-it notes so that I can easily flip back to a previous event. I don't do this with library books or borrowed books because some libraries can be quite particular about things left in their books.
Do as much or as little as you need, and what suits your purpose.