Nobody ever really talks about it, but let's face it: by freeing slaves, you'd be taking away any guise of acceptability to raping black women. How important was this particular aspect of slavery in the eyes of an average slave owner?
[This response from an earlier answer focuses on rape and sexual abuse in the antebellum South and may be of interest. I think that the first paragraph of the second comment especially might be illuminating for attitudes found at the time.