The key phrase in your post is "armchair historians". The general consensus among professional historians nowadays is that the problems caused by the invasions of Yugoslavia and Greece weren't the primary cause of the delays in launching Barbarossa (most historians now attribute the delays primarily to the weather, namely a late spring thaw and heavy rain which led to river flooding in the operational area of the western Soviet Union), and that regardless of the cause of the delay, it wasn't a decisive factor in the outcome of Barbarossa, because the 22 June launch date was sufficient for Germany to achieve its objectives before the onset of winter weather. I will point out that you are partially correct, in that Greece is generally viewed as more impactful on German preparedness for Barbarossa than Yugoslavia (because more of the units committed to Greece were redeployed for Barbarossa than from Yugoslavia), but again, neither of these campaigns are considered to have had a decisive impact on the outcome of Barbarossa by most present-day academic historians.

The narrative of Yugoslavia and/or Greece causing delays which ultimately caused the failure of Barbarossa really came from people who had an interest in attributing the failure of Barbarossa to those campaigns. Most notably of course was Hitler himself, who claimed that Italy's failures in Greece, which necessitated German intervention, delayed the invasion of the Soviet Union and prevented Axis forces from reaching Moscow and Leningrad before the onset of winter. Of course, if you look at this claim with a critical eye, Hitler had every incentive to push this narrative because it allowed him to deflect blame for his own strategic blunders, which were the actual reason German forces didn't reach Moscow in time. Similarly, Anthony Eden faced harsh criticism for the decision to commit British forces to Greece in an effort to protect Greece from the invasion, which they ultimately failed to do, so of course he also had an incentive to claim that the Greek campaign led to the failure of Barbarossa so that he could claim his actions had been vindicated.

I should note that there were other contemporary actors who contradicted this claim, most notably Heinz Guderian, who said that the campaigns in the Balkans had been concluded quickly enough to keep Barbarossa on schedule and that the delay was instead caused by the weather. Now, obviously, Guderian had his own interests and crafted a narrative that suited them (shifting the blame for the failure of Barbarossa onto Hitler and off of the military leadership, and, by extension, propagating the narrative of an apolitical, "clean" Wehrmacht), but his claims are much closer to the current historical consensus and expose that there was a divide in the contemporary understanding of the issues and that the people who were more competent in military matters knew what historians now think is closest to the truth.

The current historical consensus is that the Axis could have launched Barbarossa on or close to the original schedule regardless of the events in the Balkans, and that the primary cause of the delay was the late thaw and rain in the western Soviet Union, which kept many of the rivers the Germans would have to cross (including the Bug and the Dniester) in flood later than normal. The weather is, of course, always the elephant in the room when one embarks on the perilous enterprise of invading Russia, as the Germans found out in the winter of 1941. No amount of preparation on the Germans' part could have prevented the problems with the weather that occurred in the spring of 1941, which is why most historians consider the Yugoslavia/Greece issue not to have been a decisive factor. Yes, the need to redeploy and resupply forces from the Balkans to the Eastern Front presented logistical issues, but as John Bradley and Thomas Buell note, only 15 of the 152 divisions involved in Barbarossa were involved in the campaign in Greece, so the proportion of German forces that were affected in this way was relatively small, and Germany probably could've launched Barbarossa on time without these forces. David Stahel suggests that the need to return these divisions to Germany for resupply was costly for Barbarossa, although their impact was most significantly felt by Army Group South, rather than Army Group Center; one could extend this argument to say that if Army Group South had these forces at full strength at the beginning, it could have achieved its objectives more quickly without assistance from Army Group Center, but the decision to divert forces from Army Group Center to the south instead of continuing the all-out push to Moscow was entirely Hitler's own doing and his military advisers strenuously objected to it.

I should note that there are different interpretations of exactly what effect the Balkan Campaigns had, and some historians actually claim positive effects. David Glantz (probably the top expert on the Eastern Front in the English-speaking world) said it may have helped the Axis conceal their intentions to invade the Soviet Union by convincing Stalin that Germany's attentions were focused elsewhere, despite the avalanche of intelligence which informed Stalin that Germany was going to invade (including Richard Sorge famously giving him the exact date of the invasion, 22 June). Obviously, the Germans weren't fighting in the Balkans as a diversionary tactic; they conducted actual diversionary operations (Operation Harpune and Operation Haifisch) in the spring of 1941 alongside the actual operations in the Balkans. However, it's certainly possible that their involvement in the Balkans could have given Stalin a false sense of security about their actual intentions, since they had large numbers of forces committed away from the Soviet front (despite the ongoing buildup along the German/Soviet border which Soviet intelligence was well aware of). Stahel has argued that Hitler was cognizant of this as a potential fringe benefit of the German actions in the Balkans, although it wasn't a primary consideration.

In any case, the most important point here is that, regardless of the cause of the delays, they weren't decisive to the outcome of Barbarossa, and that the Axis forces could have comfortably achieved their objectives with better planning and better strategic decision-making. Most historians who have written about Barbarossa in recent years have argued that a 22 June start date was sufficient for Axis forces to reach Moscow by September, if not sooner. They didn't, of course, due to logistical/supply problems (the rapidly advancing mobile forces outpacing their supply lines across the vast expanses of the Soviet Union), a serious underestimation of the strength of the Soviet forces, and poor strategic decision-making (Hitler's decision to divert Army Group Center from the push toward Moscow and conclude operations in Ukraine and at Leningrad first, over the objections of his military advisers). Everything beyond that was basically blame-shifting and damage control.

This is why it's important to look at contemporary analysis with a critical, historical eye rather than taking it at face value (and why first-hand sources like Shirer are interesting but also potentially problematic). Those narratives may be desirable for everyone involved (Hitler could deflect blame, Greek/Yugoslav resistance movements could claim a significant impact on subsequent events despite their proximal failures) but may nonetheless not be supported by the factual evidence, as is the case here.

Sources:

Anthony Beevor, The Second World War (Back Bay, 2012)

John Bradley and Thomas Buell, The Second World War: Europe and the Mediterranean (Square One, 2002)

David Glantz, Before Stalingrad: Barbarossa -- Hitler's Invasion of Russia, 1941 (Tempus, 2003)

...... and Jonathan House, When Titans Clashed: How the Red Army Stopped Hitler (UP of Kansas, 1995)

Heinz Guderian, Panzer Leader (1952; republished Da Capo, 2001)

John Keegan, The Second World War (Penguin, 2005)

Robert Kirchubel, Barbarossa: The German Invasion of Soviet Russia (Bloomsbury, 2013)

David Stahel, Operation Barbarossa and Germany's Defeat in the East (Cambridge UP, 2011)