Arlington was the girlhood home of Mary Custis, her father George Washington Parke Custis was the adopted son of George Washington and also grandson in law through Martha. Mary married young Robert E. Lee shortly after his graduation from West Point.

Upon his death the lands and house passed not to Robert E Lee but to his daughter Mary and thence to her and Lee's children. Lee was named Executor of his will though and spent several years trying to fix up and make the plantation profitable as he seems to have been fond of the place too, it having been his one true home for 30 years between dismal and distant army postings.

When war came Lee and the family left for Richmond and the US Army quickly seized large parts of Northern Virginia across the Potomac to protect the capitol, Arlington was no exception and was only assured as Lee gained prominence about 18 months into the war.