I think William T. Rowe answers this question the best in his book China's Last Empire: The Great Qing. He describes the decline of the Qing as " a perfect storm of three simultaneous problems: the external shock of the expanding West, a secular crisis caused by an accumulation of socioeconomic difficulties over the long term and more acute political dysfunctions associated with the familiar pattern of the dynastic cycle." As someone else suggested, Kenneth Pomeranz Great Divergance is critical in understanding this topic. Instead of viewing China as falling behind the Europeans, it is important to understand that the European experience was very much outside of the norms of economic and technological advance. China continued to advance at a normal rate, while Europe leap forward. I would argue that this was mostly due to their colonization of other countries and lands, but that is up for debate among scholars.

Mark Elvin advanced the idea of a "high-level equilibrium trap in 1972. This theory argues that "late-imperial China's pre-industrial economy operated at such a peak of efficiency that all possible surplus had already been squeezed out of the existing technology and expended on population growth. (Keep in mind that China's population more than tripled during the Qing) Consequently, there was an active disincentive to move to a higher level of technology (industrialization) because the dislocation cost would be too great." (Rowe, 218) He also argued that the cost of maintaining the old technology was so great that investment into new technology was basically impossible. The social cost of not maintaining the old technology was also too great to bear, leading to a technological lock up.

The full answer to your question is much more complicated, and still under a fair bit of debate. Personally I find that Pomeranz does a great job challenging the traditional view of China "falling behind" Europe, which is flawed for many reasons other than its clear Eurocentric bias. I would encourage you to read the Great Divergance, and also China's Last Empire if you are really interested in this topic. I don't think a simple reply would ever fully answer this question, and there are many different schools of thought among scholars concerning this, most of which whom disagree with each other.

I like the answer of /u/myplasticlove

I just want to add a few things: There's one more very important thing about Mark Elvin's idea. The driver of industrialization have been to save labor due to higher labor cost. China don't need to save labor cost, and never had to. Her population is and was in the Ming and Qing way too large. To produce more, it would've been cheaper to hire more labor than make new machines. There was a spinning machine recorded in 1313 very similar to the spinning jenny that never enjoyed widespread use and was forgotten 300 years later.

Quoting Mark Elvin: "What China thus needed was something not within the bounds of technological possibility - namely, a machine that would produce cotton cloth with half the previous input of raw cotton rather than half the previous input of human labor".

Also, while scientifically and labor-efficiency China fell behind, overall production and trade balance China was ahead until after the second Opium War. India also did not fall behind in this front until the early 1800s. No one's really sure why Europe jumped ahead scientifically, but as China (and India) were ahead in production and trade balance even in the early/mid Industrial Revolution, I'd say in the end it fell to technology of war.

As for war technology, by 1600 the Jesuits already observed that Europeans were ahead in that front. I am going to stand beside Valerie Hansen's hypothesis (The Open Empire: A History of China to 1600) again that it is based on need. China's war technology also advanced really fast during two periods: The Warring States (massed crossbows) and the Song Dynasty (first gunpowder weapons). These two periods are marked as prolonged periods of division and constant warfare between relatively equal powers for the control of China. In contrast, the Ming and Qing were unified. Except for certain, short periods, the dynasty was never at stake.

What drove Europe's innovation in war was necessity. Between relatively equal powers of relatively equal technology level competing for domination, everyone's looking for anything to give them a bit of edge. So cannons need to be built ever better to pound in fortifications, and fortifications need to be ever better to resist better cannons. Gunpowder formula and gun technology also steadily improved in the quest to one-up each other.

China had no such use. The steppe nomads had no grand fortress that needed to be pounded in by cannon. And against a people who's way of warfare was mainly skirmishing at extreme ranges, massed crossbows can already do what early guns can (and probably better). So China had no incentive to develop better technology for war.

And really in the end it was war technology that mattered, for it was what was used to force favorable trade clauses down on the targets of colonization (not quality or quantity of goods, which were in China/India's favor).