They essentially all signed their paroles which said they would not take up arms against the Union and went home. At Appomattox the Army of northern Virginia marched out unit by unit and stacked their arms in front of Union troops (for the infantry it was actually led by General Joshua Chamberlain of Gettysburg fame). Then usually they were given a some rations for the journey, with cavalry and officers being allowed to keep their horses. Generals Grant and Sherman were quite set on forming a peace that would hold and thus were willing to hold to the idea of the magnanimous victor that Lincoln wanted where the South was beaten but would be helped up by the North.

Source: The Passing of the Armies: An Account of the Final Campaign of the Army of the Potomac by Joshua Lawrence Chamberlain

This is the actual text of the terms offered by Grant, you will notice it is very explicit that the men were free to go home. "I propose to receive the surrender of the Army of N. Va. on the following terms, to wit: Rolls of all the officers and men to be made in duplicate. One copy to be given to an officer designated by me, the other to be retained by such officer or officers as you may designate. The officers to give their individual paroles not to take up arms against the Government of the United States until properly exchanged, and each company or regimental commander sign a like parole for the men of their commands. The arms, artillery and public property to be parked and stacked, and turned over to the officer appointed by me to receive them. This will not embrace the side-arms of the officers, nor their private horses or baggage. This done, each officer and man will be allowed to return to their homes, not to be disturbed by United States authority so long as they observe their paroles and the laws in force where they may reside."