Speer certainly said something like that; Webster and Frankland's The Strategic Air Offensive against Germany, 1939-1945 includes in its appendices "Interrogation of Albert Speer, former Reich Minister of Armaments and War Production. (18th July 1945)" Regarding personnel, he said:

"I do not know the detailed figures of personnel engaged in A.R.P. and in bomb damage repair organisations, but in the year 1943-1944 the figure may be reckoned at some 1,000,000 - 1,500,000. Needless to say, this was one reason why industry was unable to make good its shortages of manpower. It is not, however, to be assumed that a materially greater output of armaments would have been attained even had the necessary manpower been available, since the bottlenecks in materials (raw steel, etc.) would still have persisted.

There is no doubt that, in the absence of air raids, it would have been possible to withdraw several hundred thousand more soldiers from the armaments industry at the end of 1943. A large proportion of German skilled labour was required at the factories for bomb damage clearance, where their specialised knowledge and keenness to restore the plants made their presence indispensable after air attacks. If no air raids had taken place, we should have been able to increase the proportion of foreign and unskilled labour.

Furthermore, during 1944 Army training units were increasingly employed on bomb damage clearance work, leading to a reduction in the standard of training and to a lengthening of training schedules."

In addition to those involved in civil defence and bomb damage repair, the air defence of Germany absorbed considerable resources. Horst Boog in Germany and the Second World War, Volume VII puts figures of almost 75% of heavy flak batteries and 70% of medium and light flak batteries devoted to home defence. For personnel, "In August 1944 the anti-aircraft force as a whole had 662,000 troops and 448,700 auxiliaries, whose numbers included 221,890 RAD-Flak, Luftwaffe auxiliaries, and men and youths of the Heimatflak, and 128,710 female flak auxiliaries and RAD 'Maidens', together with 92,000 foreign personnel including 51,000 Russian prisoners of war." Though not all AA personnel could have been released as front-line troops, the numbers were still substantial.

Regarding anti-tank capability, in Webster & Frankland Speer isn't as specific as the Harris quote (though there may be other reports that use a more exact figure), but does highlight other areas that were impacted:

"Some 30% of the total output of guns in 1944 consisted of Flak guns, while some 20% of that year's output of the heavier calibres of ammunition (from 7cm. Upwards) consisted of A.A. shells.

Between 50% and 55% of the armaments production capacity of the electro-technical industry was engaged on the manufacture of radar and signals equipment for defence against bomber attacks.

33% of the optical industry was engaged on the production of aiming devices for A.A. Guns and for other anti-aircraft equipment.

The fighting power of the Wehrmacht was considerably weakened by reason of the above, since the production of valuable Flak guns would have supplied us with excellent anti-tank weapons and the use of Flak ammunition at the Front, in addition to other types, would have provided a very substantial increase in stocks."

Exact "what-ifs" are always difficult, I can't recall seeing anything quite as precise as a 50% reduction in anti-tank capabilities elsewhere, but the broad point of diversion of both material and personnel effort to anti-aircraft defences and civil defence is borne out in post-war writing (e.g. Richard Overy's recent The Bombing War: Europe, 1939-1945).