It certainly wasn't unknown though in my era but not normal, hopefully others of latter eras can come in with a sense of numbers during those times.

The maximum size during Former Han was 3,000 ladies and officials could get rather unhappy during the time of the Latter Han or the three kingdoms when harem sizes got towards the thousand number with complaints of the cost and the Emperor's focus. In Wei Emperor Cao Rui's case, both get complaints about the excessive size of his harems (or having women working on imperial documents) and not visiting harem enough amidst concerns about lack of heir.

In the most extreme case of that era, despite Emperor Huan of the Han dismissing five hundred women to allay criticism, Xun Shuang later accused Emperor Huan of having 5-6,000 ladies and Xiang Kai (among many other charges) 6,0000 just in the third rank of concubines for Emperor Huan. During the civil war that followed Han's fall, the last Emperor of Wu Sun Hao surrendered to Sima Yan with 5,000 concubines, the Jin Emperor Sima Yan took several thousand of those ladies into his own harem along with some of Sun Hao's entertainers.

It is possible that there is some degree of exaggeration, officials rounding up and using female servants in the apartments to make their point and Sun Hao was hit with various accusations due to being Wu's last Emperor, but certainly, it could and did get into the thousands.

Also yes, any work I'm aware of covering the era, be it Fire Over Luoyang by Rafe De Crespigny, ZZTJ translation by Achilles Fang or Empresses and Consorts by Robert Cutter and William Cromwell use the term harem. People may misunderstand harems but say the word and people get the idea of multiple concubines in the palace.