The book on the Ottoman Harem is Leslie P. Peirce's The Imperial Harem: Women and Sovereignty in the Ottoman Empire, but unfortunately I haven't read it (my interests are generally later, further from the capital, and less elite).

I do know later accounts of non-imperial harems. Elizabeth Fernea's Guests of the Sheik is a book I predict you'll quite like. Fernea's husband is an America anthropologist in Iraq who's studying irrigation in a tribal village next to a "modern town" (the tribal/town distinction is a native one, and the Ferneas definitely live in the tribal settlement on the edge of town, making Elizabeth Fernea's network primarily, but not exclusively, among the tribal women, including the sheik's harem). It's an amazing book because it shows what rural Iraq was like for women. Fernea is an entertaining writer with an eye for detail--she original took notes for her husband, who had no access to women's lives, and with encouragement from him and others, she wrote them up in a book that's more widely read and cited than her husband's work (she eventually earned an academic position in her own right, and continued to study women in the Arab world). It's one of the few books I've seen that gives such access to women's lives, including the Sheik's harem (of three wives--the youngest close to Fernea's age and who we get a particularly clear view of) and their social obligations and relations. It's much later than you want, and much less grand, but as far as I know a unique anthropological document. One of my favorite scenes is when one of the older wives talks about how the sheik doesn't like women smoking cigarettes (something common in their harem socializing) and the youngest, favorite wife immediately takes a cigarette and smokes it to show she doesn't care. The book is also full of other regretful women making Fernea promise that she'll "demand gold from husband" (jewelry remains a woman's property in death or divorce) while she is still young and beautiful. It's a very nuanced portrait.

There are a few other twentieth century accounts of being raised in a harem, sociologist Fatima Mernissi's Dreams of Trepass comes to mind, though the account is somewhat fictionalized (it is based on her own experiences, but it is not purely a record of her own experiences). There are a few other similar accounts, including one of an early 20th century Ottoman example of a Circassian woman, though I'm having trouble locating the title. The Concubine, the Princess, and the Teacher: Voices from the Ottoman Harem translates the accounts of three women who lived in the imperial harem, 1876-1924, but unfortunately I've only read reviews of it. Unsurprisingly, the Imperial harem appears to have much more rigid protocol and hierarchies than the provincial harems like the one Fernea writes about.

You mention rivalry and jealousy. I'd also be interested to hear about the opposite. The idea that women in a harem would be romantically/sexually interested in each other emerges frequently in today's romantic fiction. Was this also something people thought when harems actually existed, either in disapproval or titillation?

There is a very insightful book by Emily Ruete called "Memoirs of an Arabian Princess from Zanzibar"

Ruete was born and raised in the harem of the Sultan of Zanzibar. Later in life she converted to Christianity and moved to Germany - where she wrote her memoir.

She gives a very interesting perspective on life in the harem. Her and the other girls were given education in things like knives and horseback riding. She also discusses how much power and sway the concubines had over the male figures of the house - especially the Sultan.

Also of note is the views she presents about race and ethnicity. Ruete explains that the concubines from different ethnic groups would only sit together and this created rivalries between them. The three groups most discussed are Circassians, Abyssinians, and East Africans. She also has very strong racist views about each ethnic group - for example she views East Africans as only meant for hard labour as they are not smart enough for anything else.