I just finished reading "Inside the Third Reich" last night and found it to be one of the most fascinating books I've ever read. Speer is an enthralling writer, yet also at times an unreliable narrator.
Some burning questions after reading it are:
1 - Is there any evidence he planned to assassinate Hitler?
Personally it read to me that even if he had managed to acquire some gas for his plot and all the conditions had been correct, he never would have been able to bring himself to do it.
2 - Has any evidence been brought that Speer knew about the holocaust or other atrocities, and if so how involved was he?
It seems very hard to believe he didn't know, considering how much he was in control of and how on top of things he seemed.
3 - What did he leave out, are there any stories about Speer that portray him in a different light as to how he portrayed himself?
As a person who has studied Speer quite a bit, I feel that I can provide some answers on this.
1.) No. Apart from Speer's own memoirs, there was no other evidence to indicate so- note that Speer said that he had planned it independent of others.
2.) There are two viewpoints: Yours (that he probably knew), and the other one: that he didn't know. In the case of 'he knew', the argument is that Speer as the person who effectively controlled the Reich economy (I know I know, he was only arms minister, but given his mandate and involvement, he was very big), he had known about this fact. The other argument is more along the lines of the one that Gita Sereny made, in that he had nominally known only, preoccupied as he was with the arms situation and the politicking of the party- both are possible, and the debate still continues today
http://www.theguardian.com/world/2007/mar/13/secondworldwar.kateconnolly
Nevertheless, 'how involved was he' is harder to answer. My own hypothesis is that he probably wouldn't have known about it- although it is entirely possible he had simply omitted it from his statements.