Your concerns aren't unfounded. More can be said on the matter, as always, so if anyone would like to put their oar in on the merits of Shirer's book, you're more than welcome to do so! For the meantime, OP, you may be interested in u/kieslowskifan's critique of Shirer's book, and this later expansion on same.