When Africans were first taken captive they were often sent to various forts along the coast of Africa. An example of one of these forts is Elmina Castle in Elmina, Ghana. It still exists to this day and you can take tours of this "slave factory" to get a better understanding of it. One area which can be seen is where the women who were held captive are taken to a holding cell that lay directly below the Governor's suite. From the Governor's balcony he could select a woman, extend a ladder from a trap door in his suite, and bring the woman up to be raped. Reportedly, if the woman became pregnant before she was taken on the ship she was given freedom.

In 1656 a woman sued for her freedom on the basis of being the daughter of an Englishman and an enslaved mother. Under the current laws of Pater familias she was able to stress her English heritage enough to gain her status as a free woman of color. However, the result was the passing of the Partus Sequitur Ventrem or "That which follows the womb" which transmits the legal status of a child through their mother, rather than the father. Matrilineal slavery was created with the specific purpose of solving the messy issue of what to do with the children of rape, as well as ensuring a smoother transition in the slave economy to breed the enslaved women.

One of the major justifications for slavery was an ideology called Paternalism. In short, this idea was that it was the duty of the slave owner to care for his slaves as if they were his children. He was the master and father figure of the household. Enslaved peoples were considered children. This view saw the parent-child relationship as the fundamental basis of society and formed everything else around it. Enslaved people were considered childlike, loyal, and often lazy.

The master should never establish any regulation among his slaves until he is fully convinced of its propriety and equity. Being thus convinced, and having issued his orders, implicit obedience should be required and rigidly enforced. Firmness of manner, and promptness to enforce obedience, will save much trouble, and be the means of avoiding the necessity for much whipping. The negro should feel that his master is his lawgiver and judge, and yet his protector and friend, but so far above him as never to be approached save in the most respectful manner. - Southern Magazine, 1855

Part of the justification for slavery was the idea that slaves were inherently weak minded. Enslaved peoples used this to their advantage and often resisted their captors by playing into this stereotype. These stereotypes still exist in thinly veiled racist language, but at the time it was considered a medical issue. Dr. Samuel Cartwright became famous for diagnosing "diseases of the negro" such as dysaesthesia aethiopica (emphasis mine):

From the careless movements of the individuals affected with the complaint, they are apt to do much mischief, which appears as if intentional, but is mostly owing to the stupidness of mind and insensibility of the nerves induced by the disease. Thus, they break, waste and destroy everything they handle, abuse horses and cattle, tear, burn or rend their own clothing, ... They wander about at night, and keep in a half nodding sleep during the day. They slight their work, cut up corn, cane, cotton or tobacco when hoeing it, as if for pure mischief. They raise disturbances with their overseers ... When driven to labor by the compulsive power of the white man, he performs the task assigned to him in a headlong, careless manner, treading down with his feet or cutting with his hoe the plants he is put to cultivate; breaking the tools he works with, and spoiling everything he touches that can be injured by careless handling. Hence the overseers call it "rascality," supposing that the mischief is intentionally done.

Cartwright also diagnosed a disease called drapetomania which was the disorder which caused a slave to run away.

There are multiple journal sources of white women who discuss the sexual abuse of enslaved women and how it related to this paternalism. Mary Boykin Chesnut was quoted in her diary as saying:

This only I see: like the patriarchs of old our men live all in one house with their wives and their concubines, the Mulattoes one sees in every family exactly resemble the white children - every lady tells you who is the father of all the Mulatto children in every body's household, but those in her own, she seems to think drop from the clouds or pretends so to think...

Perhaps most famously were the children of Thomas Jefferson at Monticello. Additionally, new terms came into being to describe the "mixed race" peoples - mulattoes, quadroons, and octoroons. This is not to say that the children were treated as free peoples. An example of this can be seen in civil war cartoon by David Claypool Johnston. https://www.nga.gov/collection/art-object-page.182753.html

In 1855, an enslaved 18 year old woman in Fulton county Missouri, named Celia was put on trial for having killed her owner, Robert Newsom. Celia had been purchased at the age of 14 in 1850, with the express purpose of being Newsom's concubine. She lived in a cabin just outside the main house were Newsom lived with his two daughters (one of which was the same age as Celia.) While pregnant with her third child, she asked the daughters to intercede on her behalf because she was ill, hoping that Newsom would not continue to rape her. Newsom's daughters who likely were entirely powerless themselves, could do nothing. Newsom arrived to Celia's cabin to rape her, and Celia hit him over the head in self-defense. Panicked, she struck him a second time which killed him. Celia stuffed the body in the fireplace and reduced his remains to ash. She was later caught, and the trial is fairly well recorded. Celia's lawyers argued self-defense as murder of someone who was raping them was an acceptable defense. Celia's lawyer and a few other slave owners felt that Newsom had broken the "rules" of paternalism with his act. If that was because Celia was a child, or in general, we do not know. Ultimately, even the supreme court of Missouri ruled that Celia was a murderer and she was executed. This case is perhaps the most famous, but it ruled that at least in Missouri - No, it was not illegal to rape a slave. Nor was it kept secret.

Despite the conflict that paternalism brought, sexual abuse of slaves was widespread and common, legal, and an open secret.

Sources:

Celia, A Slave by Melton McLaurin, University of Georgia press, 1991

To Be a Slave by Julius Lester, Penguin Books, 1968

American Slavery, American Freedom by Edmund Sears Morgan, Norton, 2003

Pilgrimage Tourism of Diaspora Africans to Ghana by Ann Reed, Routledge, 2015