The main problems for Speer's memoirs, both Inside the Third Reich and the Spandau Diaries are two-fold. Firstly, part of the appeal of Speer's postwar writings was that as an insider, he gives a portrait of the Third Reich's leadership and it is often an unflattering picture. Speer spends a good deal of time in his postwar writings describing the personal foibles and character flaws of figures like Goering and Himmler as corrupt and incompetent. Speer portrayed Hitler as a magnetic enigma and one whose charisma masked his demonic tendencies. This is not to say that these historical portraits are manufactured or completely inaccurate, but they often served a narrative function of embellishing Speer's self-constructed image as a seduced and fundamentally naive figure. These unflattering portraits of the NSDAP elite have unfortunate implications, especially within the German middle class social milieu in which a fundamental lack of culture and education, as well as parvenu pretensions, is a somewhat damning faux pas.

The second factor undermining the veracity of Speer's postwar writings is that he was incredibly selective about how he portrayed the Nazi period and his role in it. Speer did a good deal of researching his memoirs was done via the Bundesarchiv in Koblenz and the archivists there do not recall him asking for materials on either forced labor or the Holocaust. Bombing had destroyed a good deal of the Armaments Ministry's own archives, so Speer often had a good deal of wiggle-room with which to portray his efforts in the most flattering light as possible. He portrayed himself as an apolitical figure whose main flaw was loyalty, which led to Speer embellishing his connections to the 20 July plot and even a claim he planned to kill Hitler in 1945, of which it the only evidence is Speer's word for it. The two main assistants for his memoirs, were journalists Joachim Fest and Wolf Jobst Siedler, men who had knowledge of the Third Reich, but were not trained historians. Neither they, nor their own research assistants, visited Koblenz to check the veracity of Speer's account, nor did they consult historians (Fest's view of the profession was rather dismissive). Siedler and Fest did interview some parties who were witnesses to the events Speer described, such as Milch, but many of these contemporaries of Speer were just as interested in a postwar exoneration as Speer. More disturbingly, Siedler and Fest also consulted David Irving on various points. Although Irving had yet to write his "opus" Hitler's War, it is a considerable red flag and his dalliances with Holocaust denialism has colored his earlier works on Milch and Dresden.

While Speer's lively and intimate style might enchant lay readers, the omissions and pattern of blame-shifting is rather obvious to those aware of the voluminous historiography on the Third Reich. In his review of the memoirs, the historian Golo Mann noted that Speer's direct references to the Holocaust were often quite facile and a kind of weak exculpatory. Reviews by other German historians were likewise troubled by Speer's selectivity and weak admissions of guilt. What would have been even more damning for the likes of Mann was that subsequent research into Speer and the Armaments Ministry has shown him deeply involved in issues of forced labor and aware of the exploitation and murder of the Jews. Speer relied upon an edited version of the War Ministry's Chronicle maintained by his colleague Rudolf Wolters, which he subsequently donated to the Bundesarchiv. Nor has Speer's reputation as a miracle worker in armament's production fared very well in the historiography produced since 1970. Both Adam Tooze and R. J. Overy have shown that Speer's achievements were more apparent than real, and these findings have also been largely confirmed by the Bundeswehr's Militärgeschichtliches Forschungsamt's official history of the Second World War.

So Speer's various writings are valuable in that they preserve a specific type of postwar self-fashioning of the past. The success of Speer's memoirs in the West German readership resonated with a wide swath of the population in that it was an incredibly comforting vision of a disturbing past. Speer's exculpatory picture acknowledged the crimes of the Third Reich, but in a manner that did not command a great deal of self-reflection or recrimination. To be seduced by evil is a much safer narrative of the past than to be a collaborator, or worse, a beneficiary of Hitler.

So in short, how much should you trust Speer's Inside the Third Reich? About as far as you can throw it.