I was reading a biography of Yuan Shikai and a central point that was brought forward was that despite all of China's self-strengthening in the 19th century, China's modernized army was built up for defence but was never meant to be actually used for defence, as shown by Li Hongzhang's reluctance to engage Japan over Korea and the continual losses against Japan in the First Sino-Japanese war and against Europeans.
The book then states that Yuan Shikai never forgot that lesson and took it to heart, even when continuing to modernize the nation and the army. As such, when Japan issued the Twenty One Demands in World War I, he capitulated on most things.
So, it seems that there's a trend: Despite all modernization, the Chinese armies always remained weak and corrupt enough that they could never stand up to Japan.
Yet, contrast 1915 with 1937. A mere two decades later, after many wars in China, which I imagine would weaken it, the KMT stands up to Japan. Their armies are of course not as good as those of Japan, but they still fought them to a standstill and it was not a pushover like earlier.
Why was the Kuomingtang's army able to take Japan on and the armies of Qing and Yuan Shikai not, despite the modernization and foreign training of both armies?
The KMT didn't really succeed in standing up to Japan though did they?
1937 was a disaster both in Shanghai and Nanjing. The KMT completely ignored their German advisors in most cases, were mostly unwilling to use the weapons they had for fear of them being damaged, and were just as busy fighting the communists as fighting Japan.
I really wouldn't say they stood up to Japan in any significant way. They were quite self defeating in most of the decisions being made. Granted Japan was dealing with their own infighting which slowed their advances, but the KMT really weren't doing themselves any favours. China had way more military casualties than Japan, and a ridiculous number of civilian casualties, estimated around 20 million. That's almost the current population of Australia or Taiwan or Shanghai today.
It was really only much later on that much was gained by the KMT, but often with substantial help from the rest of the Allies. Victories in Burma relied heavily on the Allies under Stilwell, not to take anything away from Sun Li-jen et al. Japan wasn't really in a great condition near the end anyway.
In what capacity you're defining the KMT as successful in standing up to Japan that wasn't happening later in the war? From everything I've read, they were far from being in great shape to do much, even when being told specifically what to do to be effective (and then ignoring it).
I don't mean to come off as hostile, so sorry if it sounds that way. I'm just kinda not convinced about the premise of the question.
Building upon what /u/keyilan has already mentioned, I would not say that the Kuomintang was able to take on Japan over the other ruling groups in China's recent history. Rather, the Kuomintang simply decided they would make a stand, while the others decided they'd rather not.
You have mentioned individuals and groups such as Yuan Shikai and the Qing Dynasty. I would put forward that if they had resisted, as the Kuomintang did, they certainly would have been able to delay the Japanese just as the Kuomintang had - but at what cost?
Like /u/keyilan has already mentioned, the Kuomintang sustained huge losses, both military and civilian, in their prolonged campaign to resist the Japanese. And while it is touted as "War of Resistance", it was in fact, the Kuomintang that took the fight to the Japanese in Shanghai, after the Marco Polo Bridge Incident. And it was during Shanghai, and later Nanjing, that the Kuomintang's best troops were all but wiped out.
And as you know now, due to Jiang Jie Shi/Chiang Kai-shek's decisions, the weakened Kuomintang later lost power in the ensuing Chinese Civil War, post-Second Sino-Japanese War, and were forced into exile in Taiwan.
I would therefore argue that while the Qing Dynasty, or Yuan Shikai, could have well taken on the Japanese - they simply calculated the costs of doing so as being far too high, and opted not to. The Kuomintang thought otherwise, took the gamble, and it did not turn out very well for them.
This is a late answer, but I have to disagree with my colleagues here.
The Qing dynasty, for all its glory and wealth, suffered from one major flaw: whereas the government was ostensibly centralized with a large bureaucracy designed for tax collection, its military was essentially nothing more than a subcontractor for defense purposes. The Beiyang army and navy did not receive funding directly from the central government: rather, they were essentially given land, much like a feudal territory, and were to maintain their operations from the taxes collected from those territories they administered. This of course becomes problematic as what is supposed to be a national institution (the military and defense of the realm) is outsourced to a small specific portion of funding (taxation of a particular region, which is to be enforced by that military entity). Essentially, the Beiyang Army fulfilled the role of an army of mercenaries, as did the other "Armies" of Qing China.
This contrasts with the Japanese military of the First Sino-Japanese War. While they may have lacked some of the fancier equipment that the Beiyang Fleet had (for instance two large German-built pre-dreadnought battleships) the military was centralized and operated as an organ of the Japanese government, not as a military contractor. This meant many benefits, not just standardization of equipment (the Beiyang forces essentially had a hodgepodge of foreign and domestic equipment, and having enough of the right ammunition was such a problem that live fire exercises were prohibited to spare ammunition, impacting combat capability), but also a clear understanding of who was responsible for what, leading to a smoother chain of command, logistics supply chain, and perhaps most importantly acquisition of financing for military operations. The Japanese government could increase taxes using its existing bureaucratic infrastructure, as well as borrow foreign moneys through access to the financial markets. The Beiyang military did not have access to foreign markets as it wasn't the central government, and the fact they had to collect their own taxes forced them to spend resources on this as well!
Let's fast forward to 1936. Several things have changed, both in China and Japan. While China certainly experienced the chaos of the Warlord period and is currently in the middle of a civil war with the Communists, not to mention with the Japanese takeover of Northeast China and Taiwan, the Nationalist government is certainly better organized than the Yuan dynasty in matters of defense. The National Revolutionary Army is now directly subservient to the KMT government. There is growing standardization of arms, as the KMT utilizes primarily German rifles (the Hanyang 88 and CKS Rifle) and German equipment (such as the Stahlhelm). While there are certainly still organizational issues, at least the KMT has made progress in this regard. Most notably, its new German-trained divisions are fully modernized and likely able to match an equal number of Japanese troops in combat, although these units are still few in number compared to the rest of the NRA. Admittedly, a very large number of NRA troops were also controlled by generals that had large amounts of autonomy from the central government. But autonomy is a very different beast from the contractor-like nature of the Beiyang army.
Contrast this to Japan. Japanese military doctrine has shifted to specializing in a short, decisive conflict, both on land and sea. This has resulted in the frontloading of combat potential. What I mean by this is that Japanese combat units are trained with the idea that "we are bringing the absolute best possible, even at the expense of replacements." This means that matters like logistics, chains of communications, replacements of both parts and troops, and so forth are less important than victory in a tactical sense. The Japanese command chain is fragmented, with flag officers often losing control of their troops and divisional commanders seeking glory launching unsupported attacks in search of victory. For all intents and purposes, despite the strict "disciplinarian" nature of the Japanese military, the central government had very limited control over its actions.
Hence we see a pattern: while Japanese tactical victories are certainly overwhelming in the early stages of the war, as time passes the Japanese offensive bogs down and managing the large Japanese forces across two fronts of China becomes difficult. While the Japanese forces can still defeat the Chinese on a 1:1 scale, the Chinese are able to commit more troops against the Japanese in local theaters. This led to a string of defeats in 1939 where Japanese attacks collapsed against Chinese defenses, culminating in a 1939-1940 counteroffensive that was extremely overambitious and costly, given the limits of KMT military capabilities. However, the Chinese were able to push the Japanese back somewhat, which if anything changed the Japanese focus of the war. No longer would they search for a decisive battle to end the war, as they had through Nanjing and Wuhan-instead they would need to commit to a longer war of conquest, including strategic bombing of Chongqing and its industries. To do that, however, would require substantial resources-resources that were running out after 4 years of war and increasing sanctions by Japan's major trade partners. Thus Japan started planning to acquire sources of these resources... but that leads us into the Pacific War, which is another story.