but for a nation that was just a decade or two away from its shift from a feudal state ruled by Shogun to an industrial one

It wasn't exactly a decade or two since the end of the Tokugawa Shogunate; by the time of the Rape of Nanking in 1937 nearly 70 years had passed since the Meiji Restoration (1868) and the end of the feudal power structure in Japan. 70 years can mean a lot for a nation - look at how Germany in 1937 compared to Germany in 1867. Heck, in 1867 Germany wasn't even a state.

The Meiji period itself, lasting from 1868 till the death of Emperor Meiji in 1912, was characterized by Japan reforming its internal political structure and expanding abroad, becoming an Empire similar to the European powers it sought to emulate (and defend against). The Imperial Japanese Army (IJA) and Imperial Japanese Navy (IJN) were created along largely European lines (the IJA was mainly modeled after the Prussian Army whilst the IJN was modeled after the Royal Navy). During this time period one can certainly find examples of Japanese troops acting in manners that would go against contemporary laws of warfare during the engagements that they took part in, but in general Japanese forces did not act more brutal than their Western counterparts - which, to be fair, was often pretty brutal. During the international suppression of the Boxer Rebellion in China between 1898 and 1900, Japanese soldiers along with their European counterparts engaged in looting, pillaging, and raping.

Nonetheless, it is quite clear that something changed in Japan between 1912 and 1937, something that would allow for the Rape of Nanking and several similar (albeit smaller) incidents to occur throughout the Second Sino-Japanese War and WWII as a whole. The Rape of Nanking was an orgy of violence that surpassed anything that occurred during the suppression of the Boxer Rebellion not just in its size but also in the way it was accepted by all levels of the Japanese Army, government, and populous, and throughout the 30s and 40s the Japanese would not only commit war crimes against civilian populations but also against prisoners of war (something they had mostly restrained from doing in the Russo-Japanese War or in World War One).

What changed was that the Japanese military had established a firm stranglehold over Japanese politics and society. The military had been an important part of government since the Restoration, but it did not dominate Japan like it did in the 30s. Indeed, during the Taishō period (1912-1926) it seemed like parliamentary democracy was gaining more legitimacy in Japan. The Japanese Army and Navy were able to snuff out Japan's growing democratic movements, however, and aided by the chaos brought on by the Great Depression they came to control Japan in all but name only.

Similar to the Nazis in Germany, Japan's military junta sought to popularize their rule by spreading an ultra-nationalist, racist ideology. In Germany this was National Socialism, in Japan it was a corrupted form of the feudal bushido code, exaggerated far beyond what bushido originally was (basically the Japanese equivalent of chivalry) and combined with ideas of the supremacy of the Japanese race over foreigners, particularly the people residing in other Asian nations.

There was no real plan for the Nanking Massacre to take place until shortly before the city was captured. Later on in the Second Sino-Japanese War the IJA would actively plan out acts of genocide through the "Three Alls" policy - "kill all, burn all, loot all" - but that wasn't in place in 1937. However, right before the capture of Nanking and order was issued by either Prince Asaka or an underling of Prince Asaka in his name allowing Japanese soldiers to "kill all captives," a vague command that allowed for the massacre to take place. Now, I'd argue that the massacre would have probably taken place anyways - Japanese soldiers were already committing war crimes against civilian populations and POWs elsewhere in China before the order was ever issued - but it would be a mistake to say that the killings were unsanctioned. However, it wasn't a part of some formal terror campaign (unlike the later "Three Alls").

The "reasoning" behind this massacre is the same sort of reasoning the Germans had behind their war crimes in Europe - the Japanese were full of notions of racial superiority and aggressive militarism. Combined with the fact that Chinese soldiers had been putting up more resistance than the Japanese had expected (indeed, Japan had initially not planned for a wide-scale war in China in 1937 - the KMT's staunch defense of Shanghai and strong actions against IJA forces were a surprise) and Chinese resistance and scorched earth tactics led to a desire for revenge on the part of Japanese soldiers in China.

To understand what happened at Nanjing you need to back up to, at the very latest, Shanghai, 1937.

The Imperial Japanese Army was known for being a highly hierarchical organization. It was also extremely brutal and poorly organized. The general staff often had difficulty exerting control over their own armies at low levels. At low levels, discipline was enforced by physical violence, with lieutenants beating up NCOs who would beat up enlisted privates. It was a horrific business.

Also around this time, there was significant Sino-Japanese race issues. The Chinese were furious at Japanese occupation of Chinese territory, and the Japanese were furious at "Chinese insolence." Race riots and massacres committed by both sides against each other further increased tension. In the case of Japan, while they were less aware of Japanese atrocities (for instance, the massacre at Port Arthur during the first Sino-Japanese War), they were made fully aware of Chinese ones (including an incident where Chinese troops hired by the Japanese deserted and stormed the Japanese district of the town, killing some civilians). This angered many Japanese and contributed to a dehumanizing viewpoint on the Chinese, that they "needed to be taught a lesson on how to be civilized."

When the war officially started, the general perception amongst Japanese forces was that the campaign would take two weeks, ala the Mukden incident's annexation of Manchuria, and that all they needed to do was take Nanjing, the capital, and the war would be over. Instead, Chiang Kai-Shek decided to turn Shanghai into a bloodbath (or as Peter Harmsen referred to it, a "Stalingrad on the Yangtze."). Then, when they finally got to Nanjing and captured it, the war was not over. On the contrary, Chiang Kai-Shek had already relocated the government and a significant amount of Chinese war industry to Sichuan province, deep into China.

As a result, the war was not over. What you had instead was a situation of extremely frustrated soldiers, who were raised in an environment where they would be physically abused by their superior without any way to vent, now in a hostile city that they had just captured after a grueling campaign in which their friends and comrades were maimed or killed, and without the prospect of going home to Japan any time soon. Add the fact that the officers had virtually no control of their men and it was a disaster waiting to happen: for instance, the commanding officer of the Nanjing force, Iwate Matsui, noted in his journal how he was concerned about the "abominable actions" going on in the city and he worried how it would sully the name of the Imperial Japanese Army. It should be noted that Matsui was later convicted of war crimes and executed for his failure to do anything to stop the massacre.