The book is supposedly a meditation on the decline of empire as reflected in the fortunes of a fictional clan, but it's likely written sometime in the era of Kangxi-Qianlong which was a period of prosperity. Why would such a book be written at such a time and why would it elicit an adoring response from its audience? Why is there a cottage industry around the book that's survived into contemporary times?
One comparison that I can make using Hong Lou Meng (红楼梦) regarding its importance is with a very famous Western piece of literature: Dante's Divine Comedy ("Inferno").
Most of the Chinese literature that I'm used to dealing with is comprised of poetry (dynastic poetry, that is) and the early written down traditional oral stories (like those compiled by pre-Qin Master Mao in the Shijing, 诗经). Indeed, that is because there's such a large amount of material surviving via written record and form that the collection of this material is quite large. More importantly, most all of it is fairly easily legible, at least in the sense of character-to-character. But, that is not to say that it's an easy task to go about without guidance or training, because instead of being written in the vernacular of either spoken or written Chinese, these works are written instead in Classical Chinese.
红楼梦 is NOT written in the Classical Chinese style and was instead written in the vernacular. This began to emerge in the Qing dynasty (for the most part) among novels (official papers still retained the classical style). Back to my comparison with Dante, Dante too composed his work in the vulgar (Italian) as opposed to the formal standard (Latin). This in itself may or may not have been intended as a symbolization of the decline of the kingdom, but I certainly have tended to lean towards believing it was (Cao Xueqin was, after all, versed in classical Chinese). In fact, the dialogue written in the vernacular was pushed by the reformers in the 20th century as a model for a standard Mandarin (the Beijing dialect that is pushed throughout the country).
The other problem was the increasing Western influences. As opposed to China influencing the West in the past (after all, they were the middle kingdom) during the Qing dynasty there was an unprecedented amount of Western pressure. The opium wars occurred just ~40 years after the publication of the novel. Western novels were on the rise. While it may have been a period of prosperity in the history of China, in a 4000 year old civilization it did not quite stack up to precedents set.