Robert Harris's excellent account in Selling Hitler details the chain of events which lead to Hugh Trevor-Roper's misidentification of the diaries as real. While he did blunder, some really unhelpful actions by both the British publishers at The Sunday Times and the German publishers of Stern really didn't help matters.
Both newspapers agreed there needed to be secrecy, so Trevor-Roper couldn't speak to other experts about the find. Someone who could read German in the Gothic script, for example. The Sunday Times also told him at the last minute that they would need his judgement on whether they were real immediately having seen the diaries. They wouldn't wait a week while he investigated, which is how he initially thought it was going to work. This essentially created a situation where if he couldn't immediately prove they were fake, they would stand as real by default.
Actions by Stern, who had the diaries in their possession and were the ones doing the displaying, really didn't help either. They displayed the diaries alongside various other pieces of Nazi memorabilia with the implication that the diaries were from the same source. They spoke very positively about the diaries, using the techniques of a salesman. They weren't asking him if they were real, they were trying to convince him they were real, a bad idea given how much was riding on it.
The most egregious example of this was in terms of their chemical testing. Stern sent some samples of the diaries to a lab to be analysed and the results came back that they contained a paper whitener which was not available until after the war. The journalist Heidemann spoke to the man he bought the diaries from, "Fischer", about this whitener, and "Fischer" reassured him that whitener had been around for years, and had been in use long before the war. "Fischer" was in fact Kujau, the man who forged the diaries, and supposedly a middleman courier, not a printing expert. "Fischer's" reassuring message about whitener being used for years was reported back to the bosses at Stern, who basically said "Well that explains that then". All of this was summed up to Trevor-Roper at the meeting as "They have been forensically tested and found to be from the Second World War period."
Although Trevor-Roper was not really as dumb as was later implied. Almost as soon as he was out of the journalists' salesman tactics, the cogs started to whir and things started to appear out of place. Among the Hitler memorabilia was a supposed letter from Hitler to a young girl which was already known about by historians but never previously seen. The letter later turned out to be another fake by Kujau, but even if it wasn't Trevor-Roper suddenly realised, as his mind wandered watching a play at a theatre, that such a letter wouldn't be found among Hitler's artifacts, but amongst the girl's. If it was from Hitler, to the girl, it would no longer be in Hitler's possession. So the fact the Hitler Diaries were displayed alongside them couldn't be evidence of the Diaries' provenance. That illusion shattered, he started to pick at the story and found other flaws. Castigated as a blunderer, in reality he was the first person to really spot what was going on. (Another British historian, Irving, had earlier denounced them as fake without seeing them at all, but he was merely a bombast trying to shock. The fact he later said they were real just before the German Government conclusively discredited them would seem to show he had engaged in no deduction).