The Wikipedia article about Imperial China's Three Departments and Six Ministries (desktop link) which describes the administrative structure of China for much of its history mentions three "other departments" which were equal in status to the main departments, but had much less to do with actual governance. One of these is listed as the "Department of Secret Books" which is only described as being "...responsible for keeping books about astronomy and astrology." When did the Ministry of Secret Books exist, what purpose did it serve, and how did it serve said purpose?
Wikipedia's summary, while not inaccurate, is overly general. The Mishu Sheng referenced in the article as the "Department of Secret Books" is more commonly known as the Department of the Palace Library. It was the central repository of imperial records, not only astrology works but also official histories and court documents. By the time of the Southern Song dynasty, early career postings in the library also served as an important "launching point" for eminent official careers. Even earlier in imperial Chinese history, the library's intersections with the Palace Secretariat, the office responsible for coordinating official communications (and eventually the most powerful political office), meant that staff in the library could have additional influence beyond just library administration.
As for astrology and astronomy: while these would no doubt have been a core component of any dynasty's central library, at least during the Sui and Tang dynasties, one division of the Mishu Sheng was indeed the Taishi Cao or Taishi Jian, the Bureau of the Grand Clerk, which oversaw the imperial court's astrology functions. These functions extended from astronomical observations to water clock calculations. In 758, under the Tang, the Bureau became an independent agency.
A couple of good sources discussing these matters: