I participated in this somewhat contentious thread on the topic recently. Basically, the goals of popular history and academic history are somewhat at odds. Most popular history is intended to tell a good story or to bring to public light some aspect of history that the author thinks people will find interesting (and thus to sell books). The academic historian, on the other hand, is using historical sources to make an argument about what happened in the past and why. Some works of academic history are good stories as well as good history, but the two don't have to go together. There's a lot of good history out there that is quite dull. As a corollary to that, academic historians are writing for an audience that consists mostly of their peers, and the things that we find interesting will tend to be different than what the public finds interesting, due to reasons of our professional inclinations and training. That doesn't mean that academic historians can't write popular history, of course. Many academics are quite wonderful writers, and would probably do a great job of making popular history. There's at least two reasons why we don't, however:

1. The aforementioned training. Getting a PhD in a historical field takes a long time. Like, eight or more years. That's a lot of training to ignore in order to write a ripping good yarn. If you're a specialist in the social history of 17th century piracy in the Atlantic you probably know a lot of really cool stories about pirates. But so does everyone else in your field, so you talk about those stories over drinks after the conference, but write your books about heavily and minutely-sourced archival questions of the power relations among ordinary pirates in Kingston in the 1670s, or whatever.

2. Popular history doesn't count towards tenure, or for landing a tenure-track job. In order to be considered for an academic job, and in order to keep that job, we must publish work that our peers will recognize as advancing or contributing to the field. This makes the nature of academic production inherently conservative, in that the people making the decisions on what counts as "good history" (or "good science" or "good sociology," etc.) have already created work in the traditional academic style, and so are inclined to continue to honor those requirements. There are at times various movements to relax or change tenure requirements to admit greater public engagement, but these have trouble getting off the ground. The truth is that at many research universities teaching of undergrads is hardly even valued in a tenure file, with the big things that matter being teaching graduate students, research, and departmental service. Academcis' time is very limited, and so they very rarely can afford to take time to write something that might be seen as not scholarly, which would jeopardize their career. After all, even very few popular history books offer their authors an independent income, and so it's hard to risk your job on something that you know your colleagues won't respect and it's quite likely that the public won't be interested in.

Nobody loves GOOD popular history more than me! I love to assign it to students, give it to friends, recommend it to people who are interested in history but don't want to slog through tedious academic debates where the goal is to add one tiny twist on an obscure argument to one-up someone who wrote almost exactly the same book.

But. Much of it is just bad. The badness comes in many flavors. The most common is just rehashing — retelling the same damned story, often not even that well. Sometimes a good retelling is fine. But often the retelling is of something that is quite out of date, and the author hasn't made any effort whatsoever to update the story.

Sometimes there are mythological/hagiographical pop histories. These are things that just straight-up promote dumb ways of thinking about the past. They are usually quite lazy.

Sometimes there are things that are just straight-up wrong, either because they have an axe to grind, don't understand the subject sufficiently, or just don't care.

OK. So what's the answer? Well, it wouldn't hurt academics to find ways of making their work more accessible, but the "game" of getting people to read something popular is a very different "game" than is played by the academy. I think they're not totally incompatible — when I write "popular" work it tries to do a little of both at the same time, making something that is both readable and engaging for a layman but contains just enough novel insights or research that the professional will get something out of it as well. But this is tough to do. Many academics just are crap at it; they think that popular writing is "dumbing it down" (it's not, it's its own genre, and in many ways its harder than writing for an academic audience), or somehow beneath them, or just can't take the editorial criticism (academics are used to not having people tell us we can't write; popular editors edit the hell out of academic writing, for good reason). And there aren't a lot of incentive structures in the academy for doing such things (and even some disincentive structures).

The other approach is for the non-academic people who write popular histories to just do a better job. Usually this requires taking more time — you can't wrap your head around a new topic in sufficient depth to write a good book about it unless you're willing to spend several years (five seems to be a magic number there) on it. The best popular histories by journalists, for example, seem to take at least this long to write.

And the best popular histories, in my view, are ones that read the academic scholarship, talk to the experts, etc., and extract the meaning and importance from them, and impart that into a readable form. Winners all around on that model.

The above should not be read as implying that academic historians don't write wrong things, dumb things, lazy things. They do. Bad is bad. But there is a reason that much popular history has a bad reputation.

And hey, let's say it: history is a form of expertise. It is not just making stuff up or dinking around on the Internet. Would you expect a total amateur to write a great book on how to conduct brain surgery, based on a few months of glancing at books others have written about it? How about string theory? We tend to undervalue the role of expertise in the historical field, because the output of it superficially looks like, well, just "stories." So how hard can it be?

But historical narratives are a really specific type of story, and it just turns out that pretty much anyone who is going to have a chance to write a good one of those usually has to spend a lot of time on it. It's not impossible for non-academics to write good history, but it is generally impossible for non-experts to do it. The journalists who write good history end up becoming experts, of a sort, themselves. And that takes time and effort.