Why was the Union Army, for the longest time, under the command of such terrible Generals? It seems like there were alot of wonderful Brigade and Division commanders, but it seems the very highest levels of leadership were filled with very inept men. I know there were very good Generals like Grant and Hancock, and very good Colonels like Chamberlain. But goodness.. why were so many of the highest ranking Generals so.... dumb?
Well, bad compared to what? To their enemies, or to what writers with a hundred and fifty years of hindsight think they should have done?
The thing is, the US didn't have some underground vault packed full of dynamite generals that it could draw on when it needed someone to command its armies. Before the war, the U.S. Regular Army was all of 16,000 men, dispersed along coastal fortifications and the western frontier. The men it was calling on to lead armies had mostly been lieutenants and captains before the war, and this was not ideal preparation. In Lee's camp, General Ewell was reported to say that in the US Army, "he had learned all about commanding fifty United States dragoons, and had forgotten everything else." Union commanders were by and large in the same boat. Frankly, it's surprising the Union got as many good commanders as it did. Furthermore, commanding at higher levels makes very different demands on generals than lower levels. It's a commonly observed phenomenon throughout military history that men who were heroic as brigade, division, or corps commanders to suddenly become timid and clumsy when given supreme command; executing someone else's plan alongside your colleagues is always easier than conceiving your own plan and imposing your will on the situation. See Clausewitz,
“A major gulf exists between a commander-in-chief—a general who leads the army as a whole or commands in a theater of operations—and the senior generals immediately subordinate to him. The reason is simple: the second level is subjected to much closer control and supervision, and thus gives far less scope for independent thought.”
Which leads me to your comment listing the supposedly inept Union army commanders. Halleck excepted, all of them were commanding in the Eastern Theatre, usually within easy reach of Washington. The Western Theatre tends to command less attention in popular memory, and while it's debated among military historians, many (including yours truly) conclude it was ultimately the less decisive theatre. The Union generally performed much better in the West than the East, and found several good generals to command its western armies. Rosecrans, Thomas, Sherman, and of course US Grant all won their first laurels commanding in the West; Pope also rose to prominence through the very successful Island No. 10 campaign along the Mississippi River. After leaving the Army of the Potomac, Burnside and Hooker actually fought quite well in the west; Burnside made Longstreet look bad in the Knoxville campaign, and Hooker captured Lookout Mountain in the Chattanooga campaign.
While Union generals were uniformly untrained for the responsibility of army command, circumstances conspired to make the situation markedly worse in the Eastern Theatre. The main culprit is the institutional culture that took hold in the Army of the Potomac as a result of Lincoln and McClellan's differences. McClellan and the Army of the Potomac blamed Lincoln for the failure of the Peninsula campaign, some insinuating the Republicans wanted it to fail so they could push through an abolitionist agenda. Before the campaign, Lincoln had taken the initiative of appointing corps commanders under McClellan, selecting almost all generals who disagreed with McClellan's entire concept of operations for the Peninsula Campaign.
The proximity to Washington made it tempting for generals to abscond to the capital after major actions to ensure their side of the story got out first, and the public focus on the East led to greater civilian involvement in military matters. John Pope promised more vigorous action after McClellan's cautious, strategic approach on the Peninsula, and was chosen to command the Army of Virginia, with significant attachments from the Army of the Potomac. Joe Hooker will end up owing his command to his badmouthing of his commander and promising to 'De-McClellanize' the army, putting him in conflict with the McClellanites who were the backbone of the army; rewarding this kind of behavior sets a bad precedent. Furthermore, the rapid turnover of commanders made it difficult to impose good institutional culture or allow commanders to grow into their role.
There's more to it: defeatist attitudes, stemming from cultural perceptions of Southern military skill, could have undermined the army's performance; the topography of the region, with several rivers serving as natural defensive lines, may have made the Eastern theatre a tougher nut to crack than the west. However, I think the dysfunctional civil-military relationship and the lack of professional training for the high command were the most detrimental for the army's performance.
Lacking experience, training, or professional education, Eastern theatre commanders were put in a situation of partisan bickering and infighting practically designed to hinder the emergence of a great leader, facing the largest and best led of the Confederate armies.
Halleck, McDowell, McClellan, Pope, McClellan again, Burnside, and Hooker. All just not good commanders.