With all this, why were they so militarily weak during their rule?

The simple answer is that they weren't militarily weak. Militarily, the first 150 years of the Song consisted of expansion and recovery of more or less all that was China proper. (What counts as China proper? What did the Ming control that the Song, at their peak, did not? Yunnan, which was incorporated into China by the Mongols, and part of Manchuria and some territory to the northwest of the Yellow River (part of Xi Xia (the Tangut kingdom) during the Song). It is fair to say that the Song successfully reunited all of China proper.)

This expansion and reunification meant that the Song were constantly at war, but this wasn't so much a matter of being constantly invaded, but of constantly invading their neighbours.

The Song suffered two immense military disasters. The first was the loss of the north (approximately the northern third of their territory) to the Jin (Jurchen) in 1127. The second was the loss of the south to the Mongols. Between these, they were unable to recover their northern territories.

The Song-Jin wars of 1123-1126 and 1126-1127 were disasters for the Song. The Song entered these almost-fatal wars confident of victory - their aggressive diplomacy brought on these wars. However, the Song were not ready for these wars. Their last round of war against Xi Xia had been unsuccessful, with high losses (supposedly in the hundreds of thousands), and expensive. This was followed by major internal rebellions (partly driven by the economic impact of that war against Xi Xia). Still, the Song sought to finally destroy the Liao (Khitan Empire), which had controlled much of northern China when the Song Dynasty was founded. This was to be done in alliance with the Jin (Jurchen). The Song performed poorly, and the destruction of the Liao was largely achieved by the Jin alone. The Song wanted their share of the territory, based on the original planned equal contribution. This led to war with the Jin.

These Song-Jin wars, and Song military performance, was discussed by myself, u/TechnicallyActually and u/lordtiandao in https://www.reddit.com/r/AskHistorians/comments/93nr8a/why_was_song_china_struggling_to_resist_the_jin/

In summary, these wars resulted in the destruction of the main Song field armies, in the capture of the capital, and the capture of the former emperor (recently abdicated in favour of his son) and the new emperor. While the Song still held most of their territories, they could not simply bring new effective armies into existence:

Rebuilding effective armies was not possible just by conscripting peasants and giving them weapons, no matter how good those weapons were. Training and good leadership, and the discipline and motivation that can result from training and good leadership, are essential. This was the problem that the led to the fall of Northern Song in 1127. While the Song army was large, not all of the Song army was effective. The local militias were only useful for maintaining local order, and possibly for local defence, and the provincial armies were much less capable than the Imperial Army. With the destruction of most of the Imperial Army in the fighting by 1126, the Song had insufficient effective forces left to stop the Jin and defend Kaifeng.

Weapons alone are not enough. Time and training are needed.

The greatest Song military failures were political and top-level strategy. Performance at the operational level (between tactics and strategy) was often mediocre; this was not helped by the often-superior mobility of the Jin (and later, the Mongols). At the tactical level, given numbers (not necessarily superior, but it's good to not be overwhelmingly outnumbered), training and discipline, Song troops often performed well. Lacking these, they often performed poorly.

Song troops were often well-equipped, and well-supplied. Support of the army (pay, food, weapons) ate the majority of the Song budget. The army was not neglected.

The idea that the Song elevated wen over wu, culture over the military, to the point of critically weakening the army, comes from the "blame the losers" (the last emperor or two of the previous dynasty) in the official dynastic histories, as discussed in https://www.reddit.com/r/AskHistorians/comments/buzrai/what_is_the_scholarly_consensus_on_the_validity/ In particular, the charge of ignoring the military was levelled at the 2nd last Northern Song emperor (Huizong, the recently-abdicated emperor captured by the Jin). However, he was militarily aggressive, and successful in the early part of his reign (taking territory from Tibet, and successful war against Xi Xia). His great failures appear to have been political - allowing corruption and factional politics to erode effective government. In particular, factional politics often removed effective generals from their posts (even leading to the executions), and placed incompetent generals in high positions. Fear of successful generals taking over the state in a coup could be exploited to get rid of even excellent (and loyal) generals (the first Song emperor, Taizu, had come to power in such a coup). Leadership and generalship matter in war, and promotion on merit matters. However, a meritorious general supported by one faction was damaging to the others, and they were major targets. Apart from the promotion of incompetence, factional fighting often also leads to falsified reports of military readiness (and even battlefield success). This kind of thing contributed to Song failure against the Jin (and against the Mongols).

The Song recovered from their defeat in 1127, rebuilt their armies, and continued to fight the Jin. However, the Jin now controlled northern China in addition to their former territories, and also had strong resources, and access to Song military technology. The Southern Song and the Jin fought each other, inconclusively, for decades. The disasters of 1123-1127 did not recur; the Song successfully resisted.

However, they weren't able to recover the north. This can't be blamed entirely on Song weakness - Jin strength certainly deserves some credit. The Jin were strong enough to resist the Mongols for 23 years, with final defeat coming at the hands of a Mongol-Song alliance in 1234.

The Song celebrated the destruction of the Jin by going to war with their former allies, the Mongols. This was perhaps a political error (and somewhat of a repeat of Huizong pushing the Jin to war in 1123 after their destruction of the Liao). The Song fought the Mongols on and off, inconclusively, until 1268, when the Mongols began their final war against the Song. The key to the Song defences on the Yangtze was the strongly fortified city of Xiangyang; here the Song resisted the Mongols for 5 years before the city fell. After this, things went downhill for the Song, and it took the Mongols only 6 more years to finish their conquest of the Song. Again, this shouldn't be blamed entirely on Song weakness - Mongol strength deserves credit. That the Song successfully defended against the Mongols for 34 years before the final war, and that the final war still took 11 years, suggests a certain level of Song military strength. The Mongol conquest of the Jin took 23 years, and their conquest of the Song 45 years. (Interestingly, the sum of these two, the time it took the Mongols to conquer all of China, is similar to the 65 years (1618-1683) that the Qing took to conquer all of China.)

The idea of "Song military weakness" should only be seen in the context of the Song military vis-a-vis the military of their nomadic neighbors, and even then the early Song was a power to be reckoned with. As /u/wotan_weevil noted, the Song managed to reunite almost all of what we would today consider "China proper." Although the Song failed to retake the Sixteen Prefectures, they constructed a complex series of fortifications along the border with the Liao that consisted of ditches, trenches, obstacles, watchtowers, forts, and the Song even created artificial swamps. The Taiwanese scholar Liu Liyan has an article titled "Why did the Liao invade in 1004" in which he convincingly argues that the Liao invaded in 1004 because it was nervous about Song military posturing in the north. Prior to 1004, the Liao was raiding Song territory annual and not all of their raids were successful owing to strong Song defenses. The Liao decided to invade in 1004 because it was convinced that if they didn't do so, then the Song would invade them. And, contrary to popular belief, the Song army performed well and managed to check the Khitan advance.

So why then did the Song sue for peace? The Song court believed (and in hindsight this was true) that securing peace with the Liao was much more beneficial in the long run because the annual tributes paid to the Liao court was considerably less than the amount that would be needed to fund military campaigns. In addition, cross-border trade between the Liao and the Song meant a lot of that wealth would end up flowing back to the Song anyways. Furthermore, Zhenzong was not a military leader like his uncle and father (Taizu and Taizong), and there was no guarantee that he could lead the Song armies into victories (Zhenzong was personally leading the campaign at the time). For the Liao, the entire campaign was to weaken the Song and prevent a Song invasion, and in the end they secured a pledge from the Song to scale down Song fortifications. So to put it simply, neither side really wanted a prolonged conflict. The Liao was content on ruling just the Sixteen Prefectures in addition to their other territories, and the Song wanted to maintain peace, even if it had to accept a humiliating treaty.

The ineffectiveness of the Song armies afterwards is mainly political in nature. The Song policy towards the military was known as the "strong trunk and weak branches." Under this policy, the most elite military units (the Imperial Armies) was kept in the capital and regional armies were purposely weakened. This was because the Song wanted to prevent regional military leaders from usurping power, as had happened in the Tang. However, the power of the regional military governors had already been crippled during the Five Dynasties period. Looking at the fate of the last two northern dynasties - Later Han and Later Zhou, they were not toppled by regional military leaders but by commanders of the imperial armies. Therefore, the Song also purposely red-taped the Imperial Armies and divided power to make sure no single person could use them to challenge the throne.

This ended up handicapping the Song military in two ways. First, border military leaders lacked the manpower, resources, and capacity to respond real time to border threats. They depended on the court to dispatch the Imperial Armies and this process tended to be slow and cumbersome and the troops, accustomed to life in the capital, did not always perform well. Second, the Song did not fully trust their military leaders on the field. Yue Fei, Han Shizhong, and others were utilized by the Southern Song court because the court had no other choice but to use them, and they became very powerful, operating their own private armies and funding their armies independent of the court. This was exactly the type of thing the Song wanted to prevent, and the Gaozong would rather give up retaking Kaifeng and the north than to risk leaving these powerful generals on the field with their private armies (although as I mentioned in the previous thread, the threat of the two captured emperors being returned also factored into Gaozong's decision to sue for peace).

The final thing to note is that the Song fought off the Mongols for almost half a century, far longer than any other state that was conquered by the Mongols. A combination of Song naval strength, advanced gunpowder weaponry, and a topography that was not suited for cavalry warfare meant the Mongols had a very hard time conquering the Song. Their campaign into Sichuan, for example, was especially difficult, as the Chinese retreated to their mountain fortresses and forced the Mongols into protracted sieges. The Great Khan Möngke was killed while leading one of these sieges. The point people often miss is that the war between the Song and the Mongols were back and forth - the Mongols did manage to conquer territory from the Song, but as soon as they retreated the Song would retake the lost territories. The Mongols had to rely on hundreds of thousands of Chinese soldiers, led by many Chinese generals, and a newly built naval force to conquer the Song. The stout walls of Xiangyang was overcome only when the Mongols brought in Muslim siege engineers to build counterweight trebuchets. If the Mongols did not have access to all these resources, then the Song might have lasted even longer.

Hi there. While waiting for an answer, this comment by u/paintedscottishwoods may be of interest in respect of Song China's military difficulties.

https://www.reddit.com/r/askhistorians/comments/4bpcxv/_/d1bef50