Well, population decline also means there is population growth. China's population usually experiences massive declines in two scenarios - major rebellions (Anshi Rebellion, Huang Chao rebellion, Taiping Rebellion) and dynastic transitions (Song-Yuan transition, Ming-Qing transition), but the effects are never permanent. All in all, there are more peaceful years in Chinese history (suitable for population growth) than non-peaceful years (detrimental for population growth). Even with famines and natural disasters, the effects would only put a dent in the population rather than stop population growth altogether. Once the famine is over, population will recover. Another thing to note is that famines and floods typically affect only one specific region, not the country as a whole. So while the population in Shaanxi might decline due to drought, bountiful harvests in Sichuan may lead to population growth.

If you look at this graph here, you will see that China's population rose dramatically from the Song onwards as new advances were made in agricultural technology, the introduction of new types of seeds, improved irrigation techniques, and the opening up of new farmlands as a result of new crops being introduced (land formerly considered useless for planting could now be used). This was exactly what happened during the Ming and Qing dynasties (two dynasties in the imperial period with high population growths), which is what I'll be focusing on.

For the Ming, the highest estimate for population growth is 0.6% increase in population per year from 1380 to 1500, 0.5% from 1500 to 1600, and 0.4% from 1600 to 1650 (the latter figure is low due to famines, natural disasters, and rebellions). The middle estimate of growth rates is 0.5%, 0.4%, and 0.3% respectively. Martin Heijdra wrote in the Cambridge History of China volume on Ming history that population growth was probably somewhere between the high and middle estimates. Ming population growth was steady, but it grew nowhere near as fast as the Song or the Qing, and this could possibly be due to the Little Ice Age which affected agriculture. Heijdra, however, believes that the population "explosion" of the Qing was not exclusive to the Qing, and that it was a Ming phenomenon also.

For the Qing, Kent Deng listed several factors to explain population growth: encouraging internal migrations into less populated areas to open new farmland; introduction of New World crops such as maize (heavily cultivated in Northeast China), potato, sweet potato, etc., which had been introduced in the late Ming; rapid government response to disasters; and lower cost of living based on the study of prices. Due to these factors, population grew exponentially during the Qing, and overpopulation was one of the reasons why the Taiping Rebellion broke out.

By the end of the Qing, the population was already between 400-450 million, the Second Sino-Japanese War and the Chinese Civil War probably saw around ~23-28 million dead, but the Communist takeover of 1949 saw another period of population growth that is continuing to this day. Jonathan Spence wrote in The Search for Modern China that several factors such as allowing women and men to divorce and remarry, improved healthcare leading to lower infant mortality, rise in life expectancy because of better food and diet, closing of monasteries and convents, banning prostitution (which brought more women into the marriage market), and the Chinese desire for more offsprings contributed to population growth starting from the 1950s.

Here is an article that gives a good overview on Chinese population growth.