This resource might be helpful for answering this question:

https://www.library.metro.tokyo.jp/portals/0/edo/tokyo_library/english/edojoh/page3-1.html

This online exhibition from the Tokyo Metropolitan Library makes the point that the "Great Interior" (Ōoku) was not restricted to only women, but was rather the primary residence of the shogun, his female servants, the shogunal concubines and wives, and their children. Some male officials were allowed in, particularly in the company of the shogun.

You might also look at Hata Hisako's article in this book: Anne Walthall, ed., Servants of the Dynasty: Palace Women in World History (UC Press, 2008): https://www.ucpress.edu/book/9780520254442/servants-of-the-dynasty

Hata's essay clarifies that doctors, painters, workmen, and "male menials" (such as message boys) also had occasional access. She uses the phrase "inner quarters" more than "Great Interior" though both are historicallycorrect.

Hata estimates that the population of women in the inner quarters was around 500, depending on the shogun. Space was somewhat cramped and life "cannot have been easy." The women came from diverse social groups, some had been previously married, and all worked hard to try and secure promotions. They looked forward to vacations so they could travel home to see their families or go on a trip such as a pilgrimage. Some looked forward to retirement from the Great Interior while others lived in Edo Castle until their death.

One other note: although the most famous Great Interior was that of the Tokugawa shoguns (who ruled Japan from 1603-1868), most lords (daimyo) had similar zones in their primary residences in Edo and in their home castles. Similarly, almost all elite warrior residences had zones divided by gender, so this phenomenon is not unique to the Tokugawa.