The Ottoman harem is the subject of an excellent 1993 study by Leslie Peirce, called The Imperial Harem: Women and Sovereignty in the Ottoman Empire. Anyone with any interest in this topic should pick it up. Really a great book.

Peirce's study is mainly about the way in which female power was exercised in the Ottoman Empire from roughly 1550-1650, but it also aims to dispel many myths about the role of the harem and of women in Islamic society. First among these is the idea that the harem's purpose was fundamentally sexual. Harems were not just large collections of women meant only to serve the man. They were actually parallel households - just as rich men gathered large numbers of male servants, rich women did the same. In the Ottoman case, the head of the harem was the Valide Sultan, i.e. the sultan's mother. Like the sultan's household, the harem was organized into a strict hierarchy, in which access to the Valide Sultan and her favor determined each woman's place and status within that hierarchy, indicated by the size of their cash stipend. Women's status was also affected by anything which tied them more closely to the dynasty, such as bearing children or becoming an imperial wet-nurse. Women in the harem would be educated and taught practical skills before being married off to bachelors from the sultan's household. The majority of them would never have sexual relations with the sultan.

Harems (in this definition, as opposed to the technical meaning of the word, which just means the part of the house which belonged to the women) were limited to the rich elite. This is, of course, because poor people wouldn't be able to afford servants and slaves.