/u/restricteddata (as his username implies) deals with this a lot in his research about nuclear weapons, and has written about it on his blog as well as a 'Monday Methods' post on the subreddit.

​

/u/sowser and /u/cordis_melum addressed similar problems in a different 'Monday Methods' post.

This is not an uncommon problem for historians, and the solution is not particularly exciting. The answer is, we have to read between the lines, seek out evidence in unorthodox places, and leverage other means to gain access to information (FOIA requests for example, or dedicated work like the GWU National Security Archive - this goes for any topic where the typically available set of documents might exclude groups, people, topics, events, so this isn't a problem limited to historians of spycraft, diplomacy, military affairs, and so on - it's also a challenge for anyone writing on mariginalized or subaltern groups or topics that are either deliberately or inadvertantly excluded from "normal" archives.

On the broader methodological problem of this, one of my favorites is the sociologist James C. Scott - Seeing Like A State and Weapons of the Weak. There's a lot of writing on archives, access, and what is and isn't deemed worthy of preservation. If you're into the post-modernist turn (and even if you aren't), Foucault is a good critic -The Archaeology of Knowledge and the Discourse on Language. Derrida also wrote on this, but I'm less familiar with his work. Still, their point is important even if you are tired of discourse/textual analysis: the archive is not simply a neutral site for the preservation of knowledge. People (and policy) decide what is to be preserved, featured, made accessible, and so on.

That said, the information won't be "forever classified." If it's really that damaging, it will be potentially be destroyed, but more likely it will come to light eventually. It may take 50 or 100 years, but circumstances will change enough (and the protagonists will die off) so that there is less danger in revealing what happened. Don't get your hopes up though - even if the whole CIA drug sales thing is true (which I'm skeptical of - mainly because people are really bad at keeping their mouths shut), you probably won't find a smoking-gun document.

The Lincoln Assassination is a good historical analogue here - people at the time, right down to the present day, have speculated that there was some direct connection between the Confederate government and Lincoln's murder, but the historical record just can't support the claim. The best recent work is Steers, Blood on the Moon, as well as the excellent Booth biography Fortune's Fool by Terry Alford. Note that I didn't say we have conclusive proof the Confederate goverment wasn't involved - we just can't say that Davis or someone high up ordered the murder. However, we DO know lots of other things by reading between the lines - we know Booth visited Confederate operatives in Canada (who were simultaneously sponsoring other attacks on the North). We know he used the Confederate courier and information network to move around, communicate with people in the Confederacy, and aid his escape, and so on.

So, despite the fact that most of the Confederate "secret" archives were deliberately destroyed when they evacuated Richmond, we can piece together quite a bit about their operations by consulting personal papers, opposing government (i.e., US) records, newspapers, financial records, third-party government records (Canada and the UK in this instance), and on and on.

We write history about difficult or obscured topics - it just takes more work and an innovative approach to sourcing.

I tackled this question in regards to the historical enquiry of the 'Death of Edward II'. Fundamentally the views of historians had changed over time, ranging from David Humes in the 17th C, adapted slightly by Williams Stubbs in the mid 19th (in which they believed that Edward died with a red hot iron up his bum). This idea lingered on until the 1920's whereby Tout expanded on this (and the Fieschi Letter) and explained that Edward COULD have escaped though unlikely. Whereby a couple of modern historians spurred on by an article in 1970's such as Ian Mortimer, Katheryn Warner and Alison Weir are 'almost certain' that Edward II did not die in 1327, but alas fled to south wales, then travelled to Italy to become a Hermit. Though a lot of modern historians, such as Seymour Phillips, who has written extensively on Edward II has quite essentially put (not verbatim) 'we will never know - we doesn't appear in history again, so its better to assume that he is dead'.

Unless we physically remove the body which has been placed in Gloucester abbey in 1327 - we will never know the truth - it the sounding comment in which most historians conclude. As stated some historians have tended to push an agenda (which has funnily enough made them a lot more book deals) such as that Ed II did indeed survive. However others again have simply decided not to really go into depths with the topic as 'we will never know' - so historical enquiry would be somewhat fruitless.

To conclude - It depends on the historian, the subject, and their own agenda. Historians such as Ian Mortimer and Katheryn Warner in this case have tended to break the historian code of making claims when there isn't much evidence of. Yet this break from the mould have most certainly benefited them financially and their name has become a lot more known. But on a personal note, I believe to make claims on which the proof will never be established, or known, is particularly unprofessional, especially when the end clause seems to be financial and renown, and not genuine historical gain.

​

Bib:

Seymour Phillips - Edward II
Ian Mortimer - The Death of Edward II in Berkeley Castle', The English Historical Review, 120 (2005), pp. 1175-1214.

I'm speaking from the perspective as a historian and not as an expert in the fields discussed. Not everything historical is in the archives. The field of history is still debating this, but I think it is fair to say that historians are no longer speaking in absolute truths, or terms of causation (example Because situation x, the world had y). The field is becoming much more open to the idea of using terms like "probably, likely, possibly" and using sources outside the archives to piece together what we do know, even if its not the full story (and no history ever is.) So where do we turn? Again, at least in the academic world there is debate here. African Diaspora scholars are a good model though. Due to the lack of traditional archieval materials,or the obvious blatant bias in those that do exist, scholars have turned to archeology, anthropology, linguistics, oral histories and addressing the meaning in the silences in the archives. Those silences are part of our story as humans, the silences are intentional and the silence does not mean abscense. "Disposed Lives: Enslaved Women, Violence and the Archive" by Marrissa Fuentes addresses these silences head on. Using what is in the archives, such as maps, city records, and newspaper clippings. She takes oral histories and what we do know and reimagines what the lives of women who appear only briefly in the official record may have been like. In her own word from the introduction: "Dispossessed Lives constructs historical accounts of urban Caribbean slavery from the positions and perspectives of enslaved women within the traditional archive. It does so by engaging archival sources with black feminist epistemologies, critical studies of archival power and form, and historiographical debates in slavery studies on agency and resistance... this book raises questions about the nature of history." We will never have a historical or archival record of what happened to the slave women Fuentes writes about in Barbados, but with a multitude of sources she can discuss what likely took place. So the short answer? While we may never know the full truth, there are still fragments to be out there. They may be biased, but so are the traditional archives.

Ethnographic work can really help bridge the gap with so many topics. Khipus are a great example in this. Khipus are knotted strings that the Incan empire used to record knowledge, the common belief is that they were for accounting only but the Chronicles mention that they contained biographies of rulers too. Sabine Hyland managed to find ethnographic data from a descendant of a khipu maker in the highlands and this revealed a wealth of information, e.g. that the knots at the end (cayte's) are essentially the name of the piece.

There's also experimental archaeology where modern humans try and reproduce pieces from the past. This is so popular that I'm struggling to pick out standout examples, PBS' Secrets of the Shining Knight, Gillian Juleffs work with Sri Lankan steel production, and the unbelievable popularity of flint knapping are fantastic examples.

There's also a third methodology, computer simulations. The Village Ecodynamics Project looked at Native American site distribution in Crow Canyon; compared this to ethnographic, historical, and environmental data and made a computer simulation which could be programmed to look at different criteria. So how would the population look if people gave corn to those struggling during crisis? how would it look if they stopped giving corn to people who didnt reciprocate? How would it look if there was 20% more deer? etc. What's interesting is that certain projects actually gave data that resembles the archaeological data. Though of course, this isn't to be taken as evidence of anything, but it's interesting to say the least.

All these methods have problems, but then again, what doesnt?

Bibliography: https://ore.exeter.ac.uk/repository/bitstream/handle/10871/14111/World%20of%20Iron%20Juleff.pdf?sequence=2&isAllowed=y

https://www.scientificamerican.com/article/unraveling-an-ancient-code-written-in-strings/

https://www.crowcanyon.org/index.php/village-ecodynamics-project

I do a great deal of research in African American history. There is a vast amount of lost information relating to some of this history, particularly primary sources. In some cases, the “information” simply never existed beyond an oral tradition, or a stray mention in a public document or newspaper.

What I try to do first is determine if there are any other primary sources closely related or even not related at all that may contain confirmation (or the opposite), additional mentions, information, etc., and search there.

If that does not work, I try to put the information I have in context with allied events/people/institutions that are very closely related. For example, if I’m looking for information on black Methodists during Reconstruction, none of those individual churches are likely to have those records. So I go to the conference record, then the denomination’s records, and, of course to public records like deeds. (And that is just an example—there is actually good documentation on this topic, but it came to mind as a decent example.)

Often, with African American history immediately after the American Civil War, there are things that show up in histories written in the early 20th century for which there is no source cited, and may be repeated across other histories, but for which records just do not exist.

In those instances where the information may be relevant to what I am working on, I may still elect to present it, but cautiously. I’ll explain where it came from, that it cannot be verified beyond whatever source can be found, place it in all appropriate contexts. Of course, with something repeated in an earlier history that isn’t sourced, I’ll try to find primary sources to verify, but sometimes they just don’t exist.

When you work often in aspects of American history where you run in to this a lot, you simply learn how to exhaust all sources and document everything you checked, and hope that one day in the future more comes to light. You also learn how to write it so people can understand it is the best narrative that can be done with the sources available.

There are, too, some things so unable to be verified that they just stay in your files hoping for the missing puzzle piece. I have many such files.

If you’ve ever researched a lynching in the American South, or the life of an emancipated slave, or a freedmen’s community, you have probably run in to these challenges.