The Han dynasty emperors expanded the borders of the Chinese Empire due both to the threat of other tribes (barbarians, as you call them( and as a way to consolidate it's 'frontiers' or 'buffer provinces'. As the Han conquered a new tribe, they began a process of reformation, where they would educate tribes in both the language and culture of Han China. This essentially meant that smaller barbarian hordes, or ethnic groups, were slowly being assimilated into the expansive Empire. The central government saw no need to expand outwards past consolidating its borders, as China had already developed a stable, self-sustaining trade within. Launching a war is a costly venture, as you would have to fund logistics (food, weapons, transportation) and you would have to impose new laws/penalties (e.g. tax) on the population. A notable example of the stress on the population would be when Emperor Yang of the Sui dynasty launched and failed all of his campaigns against Goguryeo (one of the ancient kingdoms of Korea), leading to rebellion within that caused the eventual collapse of the Sui.

They saw no need to actively expand outwards and to conquer new found territory, as they had no reason to. Unlike what happened in Europe, the various small kingdoms surrounding China entered into a tribute-gift relationship, where the Empire and said kingdom would exchange gifts as a token of peace. It's the same reason why China didn't become larger - it saw no need to conquer these regions, instead opting to engage in peaceful negotiations. No one wants to have Mongolian land, because it's essentially a really large flat plain. The Mongols wanted out, no one wanted in. The Chinese Empire would allow various Mongol tribes to marry into the Empire, or they would set up a gifting system, to keep them at peace. The same goes for most of Eastern Russia - no one wants it. The Chinese Empire set up trade with its Eastern neighbours, namely Japan and Korea. Silk, cotton, wood, horses, and so on, but they never had the need to conquer these lands.

As to why China was overtaken by the Europeans. Westernisation, really. In essence, a large financial collapse with corrupt officials and a lax government, as well as the refusal to change their (Qing dynasty) attitudes towards international relationships. During the time, China was quite self-sufficient (and for many years it was prior, as was seen during the Silk Road trade) and had no need for Western goods. China's self-sufficiency meant that most of its military technology was not developed, in comparison to the Europeans after the discovery of the Americas where they were constantly developing their navies.

Sources:

The Rise of Modern China - Hsu, I.

Encyclopaedia Britannica: China - The Early Qing Dynasty

Organizing Through Division and Exclusion - China's Hukou System - Wang, F. L.