I think it's a combination of what you say--the research process lacking mass appeal--and the fact that, in most cases, academic history has different goals from popular history. Popular histories usually need to be very narratively compelling to hold the audience's attention, while academic history is more concerned with methodological interpretation of the source material, and academics have a lot of practice reading boring more methodological writing. I don't think the two are mutually exclusive at all, but it can be difficult to master both skillsets and balance them in a single work.

I study early Christianity, which I think makes a pretty good case in point for your question, in that the differences between popular and academic treatments can be incredibly stark. Compelling narrative, when we're talking about history, first and foremost has answers. You can sometimes see that in the questions that get asked in this sub, things like the recent post, "The three wise men in the Bible? Were they Zoroastrian?" Notice that the question is worded to expect a "yes or no" answer (I won't speculate on the OP's intent). The top comment, by u/Trevor_Culley, which has around 650 upvotes at the moment, never answers the question as it's worded. It starts with the Greek word the author of Matthew used to indicate the wise men (magi), and goes into various Greek and Roman perceptions of Zoroastrians, how accurate those perceptions were or might have been, and how we might read Matthew's use of the word magi in the context of other Jewish and New Testament sources. Early on in the answer, the historian says, "The best question is not 'Were these men truly magi?' but 'What did Matthew mean by Magi?'" That's a historian's response through and through, and a popular audience (as an aggregate, not in reference to individuals) will only tolerate so much of that, because it's not concrete and therefore not satisfying. Popular books on history tend to treat their material with a much higher level of certainty than a lot of historians would be comfortable with in an academic setting, and, even when they don't read like novels, sometimes leave out, fabricate, or overplay details to get the most emotionally appealing narrative.

In short, the divide you're noticing is that most non-specialist readers want a story with concrete and intelligible details, whereas historians are looking to paint a picture, and are just as interested in the process as the finished product. We don't always get the stories we want, which can be frustrating, but we usually get good information even when it's not satisfying. Like how you never see failed experiments in popular science headlines, but for scientists those null results can be just as informative as exciting breakthroughs.

I would add to u/MagratMakeTheTea's very helpful answer, two further comments.

First, the line between 'Academic' and 'Popular' history is complex, and much less clear than you suggest. It would probably be better to see it as a spectrum. On one end, there are specialist monographs and journal articles, which are seldom encountered by the general public, and at the other, you have things like historical fiction and historical fantasy (whether in audiovisual, literary, or interactive formats), which are highly popular but have generally not been taken very seriously by historians - though this is changing. But there is a LOT going on between these two extremes.

To start at the academic end, a significant slice of highly successful academic historians end up becoming a kind of public intellectual later in their careers, and communicating with a much wider audience. You could consider, for example, academics like Mary Beard, who have strong academic credentials, but now spend much of their time and energy producing documentaries, writing generalist books aimed at a wider audience, and getting into twitter arguments with racists. More traditional paths for academics to use to disseminate their historical ideas include writing textbooks for early tertiary or even higher-level secondary education, giving public lectures, or consulting on museum displays. These all push their way towards a more 'public history' role, and this forms one of our (many) kinds of 'middle ground' between academic and popular forms of history.

At the other end, we have a wide variety of kinds and levels of popular history production. I'd recommend Hilary Mantel's Reith lectures for an example of historical fiction that might encourage you to think carefully about the value of well-researched historical fiction to our historical understanding - though Inga Clendinnen's takedown of Kate Grenville's The Secret River demonstrates some of the problems that popular historical fiction can run into. Closer to the middle ground, you will find plenty of examples of popular history-making that come from outside the academy, but fulfil pretty high scholarly standards. One that has impressed me quite recently is Dalrymple's The Anarchy, which is quite well-referenced and uses some pretty neat sources.

Second, we need to consider the institutional structures and contexts of history-makers to understand not only how the 'academic vs popular' distinction came to be, but also how it creates a lot of different 'middle grounds' between the two.

To get a job as an academic historian, one needs a PhD, which needs to contribute an original and meaningful addition to the sum of our knowledge of a topic - or else it will not be accepted, and 3-8 years of work will likely be wasted. The 'safest' way to avoid this danger is to pick a very narrow and specific area, do some archival research, and thus find new knowledge. There are more 'risky' alternatives, but they are often discouraged by advisors (who are themselves often short on time, and risk-averse). This structurally encourages young academics to focus on tightly-focused areas of expertise, and build narrow, specialist domains of knowledge.

As they rise up the academy (if they are so lucky, and get past the exploitation of postdocs, the casualised employment, and the extremely competitive job market), young academics will likely be busy pursuing research grants and churning out large quantities of specialist material as they compete to get published in prestigious journals - which is how you get ahead in those job applications. And how do you get published? Yep, specialist original research again. By the time academics hit the top of the pile, and have time for generalist surveys, their numbers have dwindled massively, for one thing, and they have innumerable other commitments.

Popular history, on the other hand, is substantially driven by various for-profit publishing industries (whether book, television, film, and digital media), and those industries are notoriously tricky to get into, and themselves risk-averse. However, in these cases, the 'risk' is of unprofitable content, and unprofitable usually means 'boring'. Thus, engagement becomes the driving factor in their construction of history. That said, different kinds of media have subtly different priorities. For example, consider the way the Assassin's Creed and the Total War franchises market themselves through claiming that scholarly consultation is important to them.

All that said, all this structural context does create opportunities for engagement across the spectrum. Increasingly, people with History PhDs who work in adjunct or other early/mid career roles are doing huge amounts of accessible public outreach, whether in popular blogs, podcasts, youtube, or on this very forum - and sometimes that gives them a platform to publish well-researched, thoughtful, accessible popular history in book form (Hi, u/toldinstone!). That can give them an alternative income stream or career path. On a very different note, some works of popular history - I am thinking, for example, of Ken Burns' documentary The Civil War - can generate significant amounts of engagement with the academy in both scholarly and popular forums.

Hopefully some of the things I've shared suggest just a few of the ways in which the gap between scholarly and popular forms of history-making might be bridged. Yes, there is a cultural divide between academic and popular history, but I am certain that if you go to a bookstore, you will find some books in the 'History' section that are written by academics, quite well referenced, and a joy to read. Here's a couple I've been enjoying recently.

I would push back on there being such a "sharp divide." Yes, there is a lot of popular junk, and yes, there is a lot of esoteric and unread stuff. But there are also lots of people who bridge that gap from either direction: academic historians who have managed to bring rigor towards works read and appreciated by large popular audiences, and non-academics who have managed to write readable works of history that nonetheless pass academic muster. It is absolutely not the case that rigorous research is incompatible with mass appeal.

What really separates out the junk from the non-junk? And why don't academics write more popular books? First, it should be pointed out that plenty of academic historians do publish with major non-fiction presses aiming at broad audiences. What they require to do so is a) a desire to do it (for whatever reason), b) the freedom to do it (to do this while untenured on the tenure track risks being punished for it by your tenure reviewers), c) the ability to do it (writing for a popular audience, as I'll get to in a minute, is a very different skill than writing for an academic audience, and is generally speaking not taught as part of the PhD process, and so must either be cultivated individually or learned somehow), and d) the ability to select a topic that will meet that appeal, to find an agent who thinks this will be worth their time, to work with said agent to take their academic idea and turn it into something that the agent thinks will sell, and have the agent sell the book contract (phew!).

So why would an academic do this? Maybe for the clout, or the hope for more money than academic books provide. But it's a lot of work. It's much harder to write for the public than for an academic audience. Why? Because you can "get away" with a lot more with an academic audience in terms of the writing. You can use jargon, complex grammar, references to ideas that everyone in your field takes for granted, hyper-detailed accounts of things most people would find dull (like going through a memo nearly line by line), and can write sprawling tomes that people will either read or skim but more or less accept the lengths of.

When you write for a popular audience, you must take a lot less for granted, must engage with whatever their preexisting beliefs might be (however obviously wrong they are to academics), must tolerate having your work go through many, many rounds of revisions that are focused on making it more accessible to a non-academic reader, must be willing to cut your length dramatically, and generally speaking arrange the work so it conforms to the narrative expectations of a reader of popular non-fiction.

"Narrative expectations" is a short-hand way to talk about the fact that the goals of academic history and popular history are pretty different, generally speaking (no matter who writes each). Popular history is meant to sort of tell an interesting story that will conform heavily to pre-existing narrative form expectations. By this I mean the common ways in which narrative emplotment works in the West, e.g., "rise and fall," "the hero's journey," the vanquishing of evil," what have you. These are mostly literally ancient narrative forms and their use has a huge impact on the kind of story you tell.

Academic history also uses narrative emplotments (you can't avoid them if you are telling a story of any sort) but they are sometimes either much more varied (which is a way to say, less consistent and thus don't make for a "gripping story"), and they sometimes are in very different modes that your popular audience expects.

One little example here from my field, the history of science, is that almost all popular science history is in a genre we might call "the triumph of science." It's about how science is the ultimate authority for the truth, about how it casts down its foes (alternative sources of epistemic authority), about how its practitioners overcome various trials (either between themselves — one theory winning out over another — or between science and the external world — e.g., politics), and how ultimately science has "changed everything" and will continue to "change everything" onward and endlessly. This genre of the history of science dates from the early 19th century in particular (though there are variations in the 18th century), and was developed primarily by science evangelists who were trying to establish science as a credible authority over, say, church or state.

And that's fine, but that's not what the modern history of science is about at all. The main narratives you will find among academics are more like, "how a bunch of people who really didn't know what they were doing thought they knew what they were doing," or "why the right answer didn't win out," or "why scientists are really just regular people and have all of the same flaws and fallibilities as regular people." (My favorite and most literal embodiment of this is the title of a book by Steven Shapin: _ Never Pure: Historical Studies of Science as if It Was Produced by People with Bodies, Situated in Time, Space, Culture, and Society, and Struggling for Credibility and Authority_.)

Now that's a really interesting argument, and one that a lot of people find interesting outside of academia — but it's not usually why someone who seeks out popular history of science is looking for that. And it leads to really different kinds of narrative emplotments (technically it is often in what is called the "comedic" emplotment, which basically says, "the world is chaotic and irrational").

You can, of course, write such things for lay readers, and some historians of science do quite a good job of that! But you are coming up, from the beginning, against an audience expectation that you either have to take on from the beginning, or try to be very subtle in subverting.

And keep in mind that any space you spend trying to subvert an expectation and argue for having a different one is space that you can't use for actually telling a new story. This is why, for example, it is much harder for the History Channel to do anything novel than it is, say, NPR: the History Channel assumes the audience knows nothing, so every effort has to build up its own independent baseline of knowledge. NPR assumes the audience basically has a college degree, and so you can start from there. I've been on both, and the latter is infinitely easier. With the History Channel, I have to explain what an atomic bomb is, what the Manhattan Project was, what the goals were, how it worked, etc. — I never get beyond the Wikipedia-level stuff. With NPR, I can skip the Wikipedia-level stuff and jump right into my own original contribution. You can see why the latter is much more interesting and rewarding, even if it is a smaller audience.

Just to briefly touch upon the other direction: the people who write good popular histories that are not academics have two things in common in my experience. One is a desire to actually tell the true story. You might think this was assumed, but it is not: there are ones who want to tell the same story (they basically think the received version of the story is the right one and never think to question it), or they want to tell a radically unique story (which often is incorrect). Most serious work falls between these two poles (the received story is usually not 100% wrong, but it is usually not 100% right). One can absolutely not assume that people who go into this work have shared levels of intellectual integrity; they simply, plainly, sadly, do not.

Second is a healthy respect for historical expertise — they have to believe that being a historian actually requires expertise, that it should take a lot of hard work, and that other historical experts should be consulted and be part of the "community" one engages with. The best popular historians do this: they clearly view academic history as being important, they interact with academics, they share drafts and notes with them, they take feedback from them, etc. Charles Mann is a great example of this, and academics (to my knowledge) love his work as a result — he essentially sees his job as synthesizing and amplifying lots of work that academic experts have done, with a goal for telling a true story that is (to most of his readership) still quite new.

Anyone who thinks that "anyone could be a historian" and that it's just telling stories and that it's not necessary to engage with academics, etc., is probably going to write bad history. Because that mindset is either going to just recapitulate what has already come before (the received story), or it's going to gallop off down a totally incorrect path with no ability to correct for it (something I see all the time in nuclear history, where people chase down pet historical theories that neglect very basic factual impossibilities).

Anyway, the above is not a complete answer, but I hope you take away that it is not so much a matter of incompatibility, so much as it is a matter of different goals, motivations, narrative forms, and so on. There is, however, still significant overlap between the two groups.