Simply put: there is much more stylistic variation among older books than modern books. People who read mostly or exclusively modern books accustom themselves to a vary narrow range of stylistic and rhetorical devices, which they process easily and effortlessly because of their familiarity.

It would be hard for me to choose three authors more dissimilar in style (in the original language or in translation) than Marcus Aurelius, Plato, and Adam Smith. The fact that they all seem similar -- because "very abstract" -- is symptomatic of a very limited literary geography, beyond whose limits all is an undifferentiated other.

As for why they wrote that way: the idea that the quantity of "digestible" material per hour of reading is a useful or even a valid criterion of literary judgement would have struck most of the writers and readers of the past as bizarre. Efficiency was not something they looked for in their reading; nor was conciseness. On the contrary, a common word of praise, from Roman antiquity to the Renaissance to the Victorian period, was copious: a word which referred both to richness and variety of vocabulary (why say something once when you can say it three times in slightly different words?) and exhaustiveness of matter.

Edward Gibbon's history of Rome, which far from a stuffy relic was a blockbuster hit in its day, sits on my shelf in seven volumes. Finlay's history of Greece is in eight; the history of Greece by Grote is in twelve . Macaulay has six volumes of essays, not counting the four volumes of his history of England; and both were wildly popular. Could they have cut down the length? Absolutely. But why? People wanted seven volumes of Roman history and twelve volumes of Greek history and six volumes of essays. They liked reading it, so reading more of it was better than reading less. If you like chocolate cake, why have a single bite if you can have an entire slice?

It wasn't simply that people liked longer books: they liked the leisurely way that the books told what they had to tell. They didn't have a "quantity of information per hundred pages" ratio that they cared about. The reading itself was an aesthetic enjoyment, over which they expected, and wanted, to take their time -- and in which they expected, and demanded, that the authors had taken their time in crafting a masterfully written, varied, and copious work of literature.

And this unfortunately is when we have to bring class into it. Because we should remember that the people reading these wildly popular blockbuster books of the past were... well, almost exclusively very well off. Mass education is a recent phenomonon. Mass availability of cheap books is a recent phenomenon. Mass free time for educated people to read books is a very recent phenomenon. What kind of person was taking their time over Gibbon's seven volumes of Roman history (including the untranslated Greek and Latin and French and German and Italian in the footnotes)? People who had received a very elite education, probably didn't have to work for a living (or not very hard), and who in any event certainly had servants to deal with the cooking and washing and childcare. Agatha Christie was a 20th century author, and as she said in her later years, she never thought she would be so poor that she couldn't afford servants (and never rich enough to afford a car).

In other words, the people reading these books had leisure. And they chose to spend a very large portion of their leisure on books. And they had a lot of leisure to spend on these books. Why should they care about conciseness? They had time. Why should they care about a plain straightforward style? They were educated enough to follow an ornate style -- and, again, they had time to read slowly and carefully, and savor the rhetoric of a Gibbon or a Macaulay. Reading, again, was itself supposed to be an aesthetic enjoyment.

Of course, you don't have to be rich today to learn to savor twelve volumes of Grote's Greek history. You can even do it without servants. You just have to have no social life.

I'll address Meditations.

The reason it seems obtuse is because it wasn't meant to be comprehensible to you or anyone else besides its author.

There's no introduction, and the ideas presented have so little context because it was the emperor Marcus Aurelius's personal journal, most likely written while he was campaigning in Germania in the years before his death. It's filled with what historian Pierre Hadot called "spiritual exercises."

Marcus didn't explain what he was up to because he knew, and he didn't intend anyone else to read it.

Marcus Aurelius:

Despite being rich and the ruler of the greatest empire on earth, Marcus's life at this time was not particularly happy. His wife had given him fourteen children, which he called his "little chicks," in a surviving letter to his childhood tutor, Fronto. But by the time he was writing Meditations, only six were still alive, a loss that must have been hard. His empire had been wracked by a plague that killed as much as 25% of its population. The treasury was empty and he struggled to put armies in the field to fend off the enemy at the gates. His trusted friend and general, Avidius Cassius, had rebelled a few years before, causing a civil war. His co-emperor and brother-by-adoption, who he loved, had recently died of the plague. His only surviving son was Commodus, who he must have at least suspected would not live up to his standards. Marcus's health had been failing for years, and here he was, stuck personally leading a war against a confederation of German and Sarmation tribes that would not stop attacking his people.

Marcus's journal, following Stoic tradition and practice, was a place of spiritual renewal for him. He used it to remind himself of what was actually important (virtue, according to Stoicism and Marcus), to get a grip on his temper (he seems to have struggled with it) to give thanks for the blessings he'd enjoyed in life (the entirety of book one), and generally to find perspective and make himself a better person.

So what to make of the journal?

First off, get a modern translation. The translations from the 17th — early 20th century are written like the translator was trying to channel Shakespeare or the Bible. Newer translations are clearer to modern readers.

But It's actually impressive that we can follow as much of it as we can. There are multiple entries so specific that no one but Marcus could have understood what they referred to, even in his own time. So we're lucky that he dropped so many context clues. He's often quoting a few words from ancient Greek or Roman plays, poets, historians, and philosophers to concisely express what he's feeling or what he's trying to remind himself of.

Modern historians and philosophers have tracked down most of these references, and several good translations will explain what Marcus was doing or referring to, line by line, through annotations. The recent annotated translation by Robin Waterfield is excellent for this.

But to give you a quick overview, many of the entries are attempts at specific Stoic spiritual exercises. Once you know the shape of them, you'll spot them in his writing. He actually repeats about a dozen exercises again and again to center himself.

Some of these are:

Praemeditatio Malorium — or the premeditation of bad things. It involves imagining the worst scenarios you can find yourself in on a given day to ensure you're not surprised or perturbed by unexpected negative events if they do occur. It was not about pessimism, but preparation.

The people I deal with today will be meddling, ungrateful, arrogant, dishonest, jealous and surly. They are like this because they cannot tell good from evil.

Memento Mori — Marcus reminds himself of his impending death a lot, because the Stoics believed there were few better creators of context. Worried about being treated unfairly? Concerned about your legacy? Had it up to here with those squabbling patricians? No worries, you'll be dead soon enough, and then none of this will matter to you one bit. Realizing that there might be no tomorrow also makes what you do to today all the more important.

“You could leave life right now. Let that determine what you do and say and think.”

The View from Above — When you're wound up in the hustle and bustle of your life, zoom out, as the bird flys. See how small you are in the grand city you live in. See how insignificant your city is compared to the many civilizations that have risen and fallen over the millenia. From space, even the planet looks very small and insignificant. So how big are your little problems, really?

“Observe the movement of the stars as if you were running their courses with them, and let your mind constantly dwell on the changes of the elements into each other. Such imaginings will wash away the filth of life on the ground…View earthly things as if looking down on them from a high point above.”

Seeing Things for What They Really Are — The sex you're lusting after is just "something rubbing against your penis, a brief seizure and a little cloudy liquid." The fancy purple robes that cost you so much are dyed with smashed shellfish blood. The meat you're salivating for is the decaying flesh of a hog, a fish, or a cow. The Stoics believed that the things we crave can get an unnatural hold over us out of all reason. Our mind glamorizes things, and gives in to what society thinks. But if we break them down to what they really are, they start to lose their hold. So is that fancy car you're eyeing a piece of art that will impress everyone and make you finally feel whole, or an expensive collection of motoroil, steel, and wheels that will depreciate as soon as you drive it off the lot and require years of your life to pay for? You decide.

Context Is King:

So if you don't have this context, a good part of what Marcus writes about makes no sense. Is he depressed? Why is he dwelling on death so much? What's up with this guy?

If you want to understand Meditations, get a good translation with good annotations and dive in, because Marcus certainly isn't going to help you.

Further Reading:

The Inner Citadel by Pierre Hadot

To be clear from the outset, the following account is probably incomplete. In particular, I mostly consider writing in English here, not translations. I'm also mostly considering changes to the formal structure of writing over time, whereas you seem more interested in (purported) changes in content.

I suspect that to some extent you're comparing apples and oranges here. Meditations, Republic, and Wealth of Nations are all works of philosophy, written for highly educated people living in the same cultural context as the author. Dawkins is not a philosopher, and despite The Selfish Gene being an influential work within the field of biology, he most frequently writes for popular audiences - ordinary people without subject matter expertise.

If you picked up a modern work of philosophy written for a trained audience, say Sellars' Empricism and the Philosophy of Mind or Parfit's Reasons and Persons or Nagel's The Possibility of Altruism, you would most likely find them a slog, even compared to Republic (which is frequently taught in introductory level college philosophy classes). Reasons and Persons in particular piles example on top of example in exactly the way you're talking about.

Likewise, I think the "pre 20th century" cutoff is probably inaccurate. If you pick up a work of fiction from the 19th Century, say Pride and Prejudice, you're likely to find it quite easy to read. But this isn't limited to fiction. Thoreau's Walden is widely praised as an accessible theoretical work. If you are looking for something more argumentative, a lot of American abolitionist literature was written to a wider audience.

I think the interesting historical question here is why this trend of writing for a popular audience, rather than a specialized one, emerged.

The enormous improvement in world literacy rates played a large role in this process. According to Our World in Data, world literacy has increased from 12% in the year 1800 to 86% in the year 2016. Even in the well educated UK, literacy rates were stagnate around 50% until the early 19th century.

What does this simplification actually look like in practice? In this article, professor of linguistics Julie Sedivy indicates that modern writing tends to favor noun compounds over long explanatory clauses. For example, instead of writing "actions that the government initiated", one might write "government actions". This practice results in shorter, more easily digested sentences, but adds complexity in that these noun compounds can contain hidden meanings (like "house boat") that effectively function as additional pieces of vocabulary that the reader must either memorize or deduce.

Indeed the length of a typical written English sentence has decreased over time, and this was noticed as early as the 19th Century. Sherman, writing in Analytics of Literature, says:

Moreover, in the times of Hooker there were reasons even stronger for the use of long sentences, or of the unit of thought as the unit of presentation, than now obtain. He wrote for educated readers, not for the people. It was the heyday of the new learning in England. Men of culture thought little of English, deemed it an ephemeral language, and cast their works as far as practicable in Latin. There was no necessity to be instantly and completely intelligible. Though there were intellectual giants in those days, it is not clear that Hooker was easy reading even to the little public that he addressed.

There's an interesting analysis of sentence length in the speeches of American Presidents in this blog post as well.

So in summary, on the basis of the sources I'm familiar with, I would say this effect is a combination of:

• Apples / oranges comparisons between different kinds of books (e.g. academic philosophy vs popular science and biology); some books in the past (e.g. English novels) are not especially tedious. Things written a long time ago may entangle cultural differences which make them harder to understand.
• Books in the past were more often written to a specialized and highly educated audience, because fewer people were literate and those who were and had access to (expensive) books were expected to have the background necessary to understand them.
• Modern writing styles have favored the use of technical language rather than ornate successions of clauses. In part this may suggest that familiarity with each writing style is a factor in whether one finds it easier or harder to follow.

Victorian literature and golden age SF tended toward wordiness because their initial publication was paid by the word, and the publisher had a specific word length in mind. "I need 500 column inches of Commander Wanker vs The Marauders from Mars, by Thursday next!" SF was frequently published in one and two-column digest format. Fewer columns, more syllables. And it's words, not characters, so it's better to use smaller words, but it's SF and some pyrotechnical affects are expected.

These days, it's possible to not only write for the layout but to write directly nto a copy of a "live" page, which I encourage. It's amazing how much it influences my presentation to know where to aim for a paragraph break. I used to work with an early eighties phototypesetter with a word processor that made .emacs seem advanced. It would spit out copy in whatever font and typesize you wished, with a physical limit of 4 inches plus margins. Alas, it wasn't smart enough to wrap columns, so we wasted a good deal of photo paper, but I had no say in the choice of layout. That came down to looking at the biggest advertising clients and what sizes of advertisements they preferred.

It was common to need to recast sentences and paragraphs in order to smoothly flow around advertisements or cleanly fit a page.

And no, I did not consult the author. Who has time for that? And like they'd understand, amirite? I have a thesaurus right here, propping up the line printer!

At any rate, a successful serialist or essayist is expected to write for the layout as much as the publication*,* in some ways; a fully-justified column can end up looking gap-toothed if a long word forces a break, leaving a few short words to fill a large area. So the writing is edited to fit the layout and the layout is chosen to fit the writing that's expected, and if I have to cuss and fiddle to make an article fit and another author just flows, I know who I prefer!

No doubt ancient scribes had words to say about certain authors as well.

These days, it's possible to not only write for the layout, but to write directly into a copy of a "live" page, which I encourage. It's amazing how much it influences my presentation to know where to aim for a paragraph break, or how word choice will affect paragraph justification.

All of this is so transparent and so obvious that I wonder if it's something that's even considered by those who learned to type on computers.

2 spaces after the end of a sentence, always and forever! See how much better that looks in a manuscript?

We Ancient Old Things are often working within forgotten mechanical constraints that became a habit that migrated into the style guide.

When you calculate "word count", you are actually calculating the number of full and partial lines. The actual count of the literal words is something your wordprocessor can give you because some people are still paid by the word.

Line count is something that your compositor or typesetter cares about, and you don't have either; not unless you are a historical recreationist. From my understanding, you don't really even need desktop publishing software; Word will flow a template just fine.