Halleck and Hooker had a bit of a rocky history; before the war, when the two were out in California, Hooker had a drinking problem, and borrowed money from Halleck without ever paying him back. As a result of this personal history, Hooker had previously been able to work directly with Lincoln while he commanded the Army of the Potomac, circumventing Halleck, the actual General-in-Chief.

Before he was actually replaced, Hooker had already tallied up a few marks against himself. Lincoln had suspected that Hooker had worked to undermine his predecessor, Ambrose Burnside, out of ambition to command the army. Writing to Hooker, Lincoln called this a great disservice to the country and to a meritorious brother officer. When he did take command and open the campaign, the result at Chancellorsville had been a humiliating defeat, so by the Gettysburg campaign, he was already on thin ice. At that time, he did not further endear himself to Lincoln by initially seeking to operate against Richmond while Lee's army was advancing into Pennsylvania.

The controversy that finally ended Hooker's army command involved the troops garrisoning Harpers Ferry; Hooker wanted them subordinate to his command, while Halleck wanted them to remain in place. When Hooker tried to issue orders to the garrison, Halleck immediately countermanded them; incensed, Hooker wrote to Lincoln, demanding the garrison and offering his resignation if it wasn't given to him. To his surprise, his resignation was accepted.