The Japanese army actually was similar in character in both battles. They were very mobile for an infantry force, had high morale and were on the whole well led. Commanders at Mukden and Khalkin-Gol (Hōten/Houten and Nomonhan respectively) both employed a similar operational plan, double envelopment.
The difference was the enemy, and perhaps the changes in military technology over the 34 years between the battles. The Russians at Mukden had poor morale, and a command that was riven by personal differences. At Khalkin-Gol, the Soviets were ably led by General Zhukov who was later to prove successful against the Germans in the west.
Moreover, the Red army had a highly advanced doctrine in 1939, based on massed firepower and mechanized mobility. This doctrine was well adapted to the improvements in machine gun and artillery fire which took place between 1905 and 1939. Additionally, Soviet doctrine made good use of armored vehicles, a relatively new technology, and mechanized logistics. This Soviet doctrine employed firepower to stop the Japanese cold during their initial offensive, and mechanized mobility to overwhelm the Japanese during the Soviet counteroffensive.
In contrast, the Japanese Army doctrine emphasizing spiritual power, morale, night attacks and infantry forces was not well adapted to firepower that WWII era armies could muster in open country.
Sources:
Drea, Edmund "Japan's Imperial Army: Its Rise and Fall, 1853-1945"
Drea, Edumund "Nomonhan: Japanese-Soviet Tactical Combat 1939"
Japanese perceptions of Soviet Force capabilities in the lead up to the battles of Khalkhin Gol are also of particular interest. While prevailing attitudes regarding Soviet competency were in no small part shaped by previous success in the Russo Japanese War and subsequent institutional mythology, Japanese experience against the Red Army in the years leading up to the decisive engagement at Khalkhin Gol greatly influenced the approach taken by the Japanese leadership in Manchuria as well. The Imperial Japanese Army performed well against the Red Army during the Siberian Expedition from 1918-1922, and although these forces were eventually forced to withdraw due to mounting pressure from the mainland government in Tokyo, the endeavor did nothing to dispel institutional notions of superiority over a potential Soviet adversary. Furthermore the experience of Japanese troops at the battles around Lake Khasan seemed to show the leadership of the Kwantung Army that Japanese soldiers with capable leadership and fighting spirit were capable of standing up to their increasingly better equipped Soviet counterparts. The experiences of the Kwantung Army against Soviet forces through this interim period did not lead the force to undertake major, systematic changes in its approach to future combat with Soviet forces, nor did it diminish its desire to engage Soviet forces in a decisive confrontation over the contested border between Manchukuo and the USSR.
Why was the Japanese Army able to defeat Russian forces at Mukden, but unable to defeat Soviet forces at Nomonhan? Above all there were clear disparities in independence of command and political support for the endeavor among the national leadership of the two countries involved.
Japanese involvement was characterized by the early initiative and instigation of events by local actors (on both sides to be clear) on the Japanese side this originated specifically with the Kwantung Army followed by its reigning in by the military and civilian leadership in Japan. The actions of the Kwantung Army were then on many occasions moderated and ultimately brought to a conclusion by the Imperial Government asserting its control over the Kwantung Army through diplomatic means or control of Army reserves once events had escalated beyond the Kwantung Army’s scope of control and capability such as the imminent threat of wider conflict. As the border confrontations increased in scale the Kwantung Army found itself increasingly reliant on the Imperial Government for the resources necessary to conduct large scale, high intensity operations, and suffered an according decrease in its autonomy and freedom of action as the Imperial Government asserted its intent to limit the scope of confrontation with the Soviet Union. As demonstrated in the preceding battles at Lake Khasan in July-August of 1938 the local success of Kwantung Army forces was halted at the point of wider escalation by the Imperial government, then subsequently erased when under the terms of peace they were forced to withdraw from the territory they had taken. Thus the Kwantung Army in its effort to seek a decisive resolution to the border question, embroiled itself in an enterprise which above anything else depended upon the Kwantung Army winning a direct confrontation of wills with its own national leadership in order to free the material and reserves necessary to match inevitable Soviet escalation in the confrontation it desired. This conflict stemmed ultimately from the desire of the national leadership in Japan to avoid a full scale conflict with Russia while Japan was bogged down in China, as well as the opposition to a prolonged land war from the Southern Strike faction.
Conversely, state support for Soviet military efforts greatly increased over the course of the Soviet-Japanese border conflicts. The poor Soviet showing at Lake Khasan caused a further stiffening of Soviet resolve and an increased desire to achieve a decisive resolution to the escalating series of border skirmishes. The posting of Zhukov as a troubleshooting inspector, then commander of the 57th Special Corps in May 1939 was a direct result of the Soviet desire to achieve a rapid and decisive resolution to the border question. It is particularly telling that on 19 July the 57th Corps was reorganised into the 1st Army Group and then given operational independence from the overarching Front Group of Soviet forces (G. M. Shtern). Soviet forces had thus been organized under considerable leadership talent which, freed from local interference was to prove capable of achieving the decisive resolution required of it by the Soviet State.
Zhukov provided a false opportunity for the Kwantung army to play out the scenario of Mukden once more and the Kwantung Army committed full heartedly to achieve the goal that had escaped it at Mukden. Much like at Mukden the Japanese encountered what appeared to be a timid, clumsy, inflexible, defensively oriented Soviet force which would be enveloped and destroyed by the experienced, tactically well led Japanese force. The historical result makes for fascinating reading.
Some other brief conclusions: Unlike Japanese Forces at the battle of Mukden the Kwantung Army was not fully supported by the military and civilian national leadership of Imperial Japan and on many occasions it found its range of actions and capabilities directly restrained by pressure from Japan. At the decisive moment Kwantung Army forces were unable to match the Soviets in escalation of forces and were decisively defeated.
The Kwantung Army forces at Nomonhan faced well led, well equipped Soviet and Mongolian forces with considerable operational independence and support from higher political leadership. These forces had the political and material support to significantly escalate the severity of the skirmish at Nomonhan in order to achieve a decisive outcome. Furthermore Soviet doctrine had evolved over the preceding years to become an all encompassing, holistic approach to warfare at all levels, emphasizing mechanization and combined arms coordination across all branches, G. Zukhov the commander of forces at Nomonhan was a highly skilled proponent of this new generation of military thought and a far more capable adversary than many the Kwantung Army had previously faced.
The Japanese forces of the Kwantung Army were unable to reconcile their previously held vision (built an admittedly long base of experience) of Soviet force capabilities with those of the force they faced in 1939 at Nomonhan. Lessons learned from the alarmingly fierce and escalatory battle at Lake Khasan in the summer of 1938 were selective and did not deter the Kwantung Army from engaging Soviet forces at the Khalkhin river. Indications of increasing Soviet resolve, tactical proficiency and material superiority were noticed too late to be compensated for by increasing Japanese Army material and capabilities and were sometimes dismissed as things that could be overcome by elan and the indomitable fighting spirit of the Japanese soldier.
Unlike Russian forces at Mukden, Soviet forces at Khalkhin Gol could rely on lavish logistical support was with a fleet of 5000 or more trucks sustaining Soviet front line positions 650km away from the nearest railhead on expanded capacity railways. This logistical support backed up a qualitatively superior artillery park which by some estimates doubled the number of guns brought by the Japanese forces and also enabled operations by large numbers of qualitatively superior tanks and armored fighting vehicles which were ideally suited to the rolling terrain around the Khalkhin river. Japanese troops operated with comparatively few trucks (yet to find a solid number) while over 300km from their nearest supply point, leading to serious supply problems that were never resolved.
While Japanese and Russian troops were relatively evenly matched in firepower and mobility at Mukden, by Nomonhan there was a clear disparity between the two forces. Kwantung Army forces were severely hampered by their lack of armored support and Japanese armor was not used en masse like its Soviet counterpart, instead being expended piecemeal for support and local counterattacks which contributed to the force’s inability to break Soviet encirclement and avoid its subsequent annihilation. By Nomonhan there was a large disparity between the firepower and mobility of the two forces which proved decisive in the open grassland around the Khalkhin River.
Shusuke Takahara “America’s Withdrawal from Siberia and Japan-US Relations,” The Japanese Journal of American Studies, No. 24 (2013)
(not entirely objective but illustrative nonetheless) V. Parfenov The Intervention in Siberia (New York: Workers Library Publishers, 1941) http://fau.digital.flvc.org/islandora/object/fau%3A4401/datastream/OBJ/view/The_intervention_in_Siberia__in_1918-1922_____.pdf
"Military Transformation: The Japanese Army during the 1920s and 1930s" by Dr. Edward Drea" https://youtu.be/r4kx4ievqSk
Geoffery Roberts Stalin’s General: The Life of Georgy Zhukov (London: Icon Books, 2012) https://books.google.com/books?id=OOxRrfalHJcC&pg=PT61&lpg=PT61&dq=57th+special+corps&source=bl&ots=GPdTWYaUAt&sig=vMa6SE0AXm0I1YZrtLzXd9cF26w&hl=en&sa=X&ved=0ahUKEwi2wfrN3unNAhVG5mMKHVyEDXUQ6AEIRTAG#v=onepage&q=57th%20special%20corps&f=false
Edward J. Drea “Nomonhan: Japanese-Soviet Tactical Combat, 1939,” Leavenworth Papers, No. 2 (January 1981) http://usacac.army.mil/cac2/cgsc/carl/download/csipubs/LP2_NomonhanJapanese-SovietTacticalCombat_1939.pdf