Lee was really loving to his children. He spolied them when they were young, contrary to the methods of the day in his region. But Grant was the same and from a different part of the country, and younger than Lee, so it's possible they represent a shift. He was known for having read a number of books on child-rearing.

Although they had a rocky start he and his wife had a loving relationship. Although, he very much liked to flirt with women, and at public functions would pay them an inordinate amount of attention. He was a perfectionist, and this got him in trouble with his wife, who was not. And until they learned to divide their spheres, they fought. All 7 of their children lived to adulthood, which was rare. I believe the daughters never married, also rare. But he was very attached to them, and it is possible that he made courting difficult for them, on account of fame, and also protectiveness. It has to be said the Civil War was a very traumatic event for all concerned. Once they were oldrer he was very keen to "direct" them to good places in life. Peck. Peck. Although most of them responded well to this and remarked in later years on his good advice and how they treasured his letters and guidance.

He was witty in private. And some considered him vain in reference to his appearance. He was very religious, and didn't believe in reading popular fiction for pleasure. Rather he suggested classic literature and the Bible. He was however a fan of Cervantes.

In public, and in his correspondence to anyone but closest family and friends he was very reserved, almost cold. And he was well known for this as well. Especially his reservedness. He was also old fashioned in his manner of written address, and prolific in his writing. He enjoyed correspondence.

He was obsessed with paying debts. He had never really known his father, who left the country when he was 6 and never returned. Light Horse Harry Lee had been a war hero and politician, and Washington's right hand man on the field, but he was a failure at business and drove the family and their friends into financial ruin. Lee was raised by his mother to not be a swindler and to cleave to his integrity as if his life depended on it.

Most sources both direct and indirect show that his integrity was important to him, that he neither impolite nor excitable, nor rash, that he delighted in his home life, and he was relatively charming to be around in private company.

Sources for further reading:

The Recollections and Letters of General Robert E. Lee : Robert Edward Lee (Son)

Lee the Last Years, by Charles Bracelen Flood

And Reading the Man: A Portrait of Robert E. Lee Through His Private Letter by Elizabeth Brown Pryor