[deleted]
Here are some of the tricks I use, outside of critiquing the argument itself:
Look at the source--the journal/magazine/newspaper or the publisher. Is the journal peer-reviewed? Is it a predatory (pay-to-play with few to no standards) journal? Is the book publisher a university press? That's not perfect, because some of the top academic publishers do not have "university" in their name (Brepols is HUGE for medieval, for example). Some university presses are better than others. Some academic publishers will let occasionally let just about anything slide, even if they are usually reliable! But it's a good place to start.
Look for reviews online, or citations to it via Google Scholar. H-net.org has a lot of book reviews that you can read for free. Books that straddle the academic-popular divide might be lucky enough to get a review in the London Review of Books or New York Review of Books. Often you can preview the first page of a review if it's in a journal catalogued by JSTOR, Project Muse, or Cambridge.
Who is the author? What is their academic background/training (if they have a PhD, where from?), what kind of general reputation can you gather they might have via Googling?
Look at the footnotes and/or bibliography (not all articles have a separate bibliography at the end). There's no general rule for what is good here, because on some topics the scholarship is old. In general, though, you want to see more recent secondary works cited--the more recent the better--and the use of primary sources as well. Overreliance on just a couple of sources (unless the article's point is to explore one specific old text or something) is a concern.
One thing that I, personally, like to see in a book or article (even if in footnotes) is a discussion of the limits of the primary sources. What they can and can't tell us, the difficulties, etc. Not all good scholars and scholarship do this, especially in articles, but many do. It should not be a dealbreaker. However, someone who is playing fast and loose with their interpretation of the sources is probably not going to explain that.
Also, if the book/article contains a review of previous scholarship, most historians try very very hard to be equitable or fair even to the people they vastly disagree with. A question of respect.
Hopefully some other people will weigh in. I'd love to get some new tricks.
Footnotes! /u/sunagainstgold is really right. Every history book, academic press or not-academic press, will have footnotes -- and lots of sloppy research gets tucked away in the footnotes, so it's a pretty good place to look. Some questions to ask:
A lot of authors drop their discussion of secondary sources and historiography in their footnotes (if not in the introduction or the body of the text itself). Do they engage with other scholarship? Recent scholarship? You want an author who knows what else has been written about their topic -- not someone who thinks they're the first person to write about 17th century underwater basketweaving.
Are there primary sources? If there aren't any primary sources, it's definitely not worth your time.
What about the variety of primary sources? Is the whole book based on one set of documents, or did the author use different sources?
Most importantly for me, could the primary sources used reasonably support the author's argument? For example, a book about British and Indian experiences about decolonization should have sources from both Britain and India, or (if the author didn't go to India) sources that represent the Indian perspective. Otherwise, how can the author make any claims about the Indian perspective?
I wouldn't get too fussed about the academic/popular press divide. It's a good way to get a first impression, but academic books can be garbage and popular books can be gold, so I'd never dismiss anything out of hand.
There are, of course, some exceptions to the rules as (very well) set forth by sunagainstgold and cookies. Personal opinion follows.
For example, my field of particular interest in Military hardware. Some of the best in the business are not full time historians. Doyle (German armor) was a draughtsman, Zaloga (Soviet armor) a policy analyst, Hunnicutt (US Armor) was a metallurgist, if I recall. And so on. They would spend their weekends or retirement doing their research. but then, you have the fact that they are doing it for love of subject matter, out of their own time and money. Further, the subject is fairly "hard" history, with little requirement for interpretation or conclusion. Indeed, Doyle/Jentz books specifically avoid it. In the introduction of some,they outright say "The following is sourced from primary sources. If we didn't see the primary source, it's not in this book. They are presented as they appear, without comment, it is up to you to infer what you choose"
Due to the nature of the subject, the footnotes rule does not apply. There is not a single footnote, or "bibliography" or "references" section in anything I've seen by Doyle/Jentz or Hunnicutt. I'm finishing up my first Hunnicutt type book and have taken a relatively similar position. I may make reference in the text that a quote was taken from this test report or minutes of that meeting, as do D/J or H, but the entire premise is that everything is sourced from government and manufacturer's documents, straight from the horse's mouth. For something like that, the validity of the book is pretty much down to the question of "just how much do you believe that the author won't make something up? What's the percentage in it?"
Just how affirmative are the statements that you are reading? If the book says "The project to give the Sherman Tank a high velocity gun started in 1941", that's something pretty damned ballsy to say without solid basis to say it. On the other hand, something less definitive like "the Lack of a high velocity high velocity gun on Sherman wasn't felt until 1944" is on the face of it fairly much a result of interpretation, and some backup statement as to -why- that conclusion was reached is required, be it in text or footnote. Is it provided?
At the risk of self aggrandisation, I'm going to use myself as an example. I'm in an interesting position as a relative newcomer to the history scene. My background is in both the military and IT. My training is in law. I've not published a single book. Yet somehow over the past four years I've earned a reputation as generally knowing what I'm talking about, and I get cited. Evidently public enough that someone felt it worth putting a "verified" tag next to my name. How did they determine my "validity?"
Near as I can tell, there were several factors at play. Perhaps all fallacies. For example, the "appeal to authority." I'm fairly unique in the area of armor history that I am paid for it as a full time job. That counts for something, right? Is it because I'm accurate, or is it because I'm good at duping my employer? Or perhaps because i have in the past been publicly associated with folks like Zaloga and Doyle? Maybe I'm just a nice guy to hang around and chat with. I publish a lot of videos and articles on the web. But I don't put a single footnote on them. (Though I will occasionally put scans of documents up as evidence when pushing something controversial).
Thinking about things rationally, there is no particular reason to believe the validity of much of what I put out. I guess one just has to go as much with "gut reaction", overall tone. I do try to separate facts from conclusions from opinions. Hopefully this helps. I talk about things which I announce are from, and unless I'm downright lying, can only reasonably come from, primary documentation. And I'm willing to acknowledge when I've screwed up or missed something.
Any historian is going to have a first appearance. And if it's on a subject like 17th century underwater basket weaving, it may very well be that his first appearance will also be the first assessment of a particular subject, which means that peer review is going to be very difficult. And someone is going to have to be the first to take sufficient of an interest in 17th century underwater basket weaving to read the article or book, so googling for reviews of this unknown author will not exactly be helpful either. Ultimately, the best thing you can do is just analyse the writing (or video) with your own smell test.