Part of the issue was the Civil War Amendments. The Union wanted the Confederate States to be forced to accede to the 13th, 14th, and 15th Amendments when they came back into the Union.

If you treated them like they'd never left, that would leave 13/14/15 in a sort of limbo, where they hadn't had a sufficient number of States to ratify them.