What were the most important reasons in the failure of the Taiping to overthrow the Qing during the rebellion of 1851 to 1864?
As for any question this big, there are likely a lot of factors, but I can concentrate on a big one - the complete breakdown in Taiping leadership.
Quick introduction for anyone not familar with the Taiping Rebellion: it centered around a scholar named Hong Xiuquan, who after he failed his civil-service exams had a nervous breakdown and dreamt that he was ordered by his father and older brother to rid China of demons. After learning more about some limited aspects of Christianity, he interpreted the figures in his vision as God and Jesus, and figured he must be Jesus' younger brother. He ended up preaching and his converts eventually began the practice of iconoclasm and spirit possession, which played quite well to the Hakka minority in Guangxi and eventually grew into a full-on rebellion against the Manchu Qing Dynasty.
In 1851, after a successful uprising, Hong declared himself "Heavenly King," with the obvious challenge to the Qing emperor that implies. He also granted the title of "King" to his five most important subordinates:
For awhile, the leadership worked quite well together, although there was some strain from the fact that Xiao and Yang, when posssessed by Jesus and God respectively, technically outranked Hong and could justify overriding him. The real problems began to take root when Xiao and Feng were killed in battles in 1852, especially since they had been extremely close and loyal to Hong, and their presence helped smooth out differences in leadership. In 1853 the Taiping captured Nanjing, which ended up being the high point of the rebellion.
Between his position as commander of the army, his spy network, and being able to make declarations as God, Yang had become the most powerful leader in the Taiping, despite Hong still holding the title of Heavenly King. Relationships between the leaders had begun to break down, and Yang decided to make his move. In 1856 Yang made a coup attempt, but Hong was tipped off and had his loyalists (Wei, Shi, and another general named Qin Rigang) return their armies to Nanjing and quell it. Wei and Qin arrived first and killed Yang, along with his entire residence. The two then took advantage of the situation and began purging all of their political opponents, killing thousands of people.
Shi Dakai arrived later that month, and reacted strongly to the purge, reprimanding Wei. When Shi left the city, Wei responded by murdering Shi's family. Shi rose his army in rebellion, and Hong eventually sided with Shi, resulting in the death of Wei and Qin. This meant that four of the original five subordinates under Hong were now dead, and Hong began to fear that Shi's extreme popularity might lead him to eventually rebel as well. Hong appointed his relatives to the newly open leadership positions, and Shi eventually took his army and left Nanjing rather than risk another bout of civil war. Hong's new leadership, as one might expect from persons hired for their family connections and loyalty rather than merit, proved ineffectual. Hong himself also began to grow more and more withdrawn and illogical, and began ignoring his military advisor's advice. When food supplies in Nanjing began to run low during a Qing siege, he merely stated that God would provide them with manna, and began eating weeds, which made him sick and eventually killed him weeks later. Nanjing fell shortly thereafter, and the Taiping were as good as finished.
Source: Jonathan D. Spence's God’s Chinese Son (1996)
/u/Magnusblitz did a good job providing an explanation on the leadership problem of the Taiping, though in my honest opinion I considered this to be overblown. I would like to make some additions and clarifications on the leadership issue.
Shi Dakai took with him some of the best Taiping troops when he departed Tianjing (Nanjing). Shi himself was a military genius and he was a renowned figure within the Taiping Kingdom for he was considered to be upright and just. Li Xiucheng, Shi's contemporary and a military genius himself, considered Shi's departure to be devastating, as it deprived the Taiping of not only an abled military commander but also key troops needed to stop the Qing advance. Shi's campaign can be considered a tactical failure for the Taiping. While it was true that he fought six years throughout China against much larger Qing forces, he did not manage to accomplish anything that would contribute to the survival of the Taiping regime and in the end his small army was defeated in Sichuan.
Wei Changhui's brother, Wei Jun, surrendered to the Qing as a direct result of the Tianjing Incident. He later played a key role in the recapture of Anqing.
Hong's new leadership, as one might expect from persons hired for their family connections and loyalty rather than merit, proved ineffectual.
This is actually not true. Yang Xiuqing was indeed a capable administrator and strategist, and his contribution to the success of the Taiping regime should not be understated. Yet his death did not contribute to the fall of the Taiping regime. Yang Xiuqing and Wei Changhui's death and Shi Dakai's departure signalled the end of the first generation of Taiping leadership, but they were quickly replaced by a new generation of politicians and military leaders who proved just as, if not more, effective as the first generation.
Hong Rengan, Hong Xiuquan's cousin, was a capable administrator who attempted to centralize power, make alliances with Western nations, and revitalize the Taiping's military successes. Li Xiucheng and Chen Yucheng were both extremely capable military commanders who led the Taiping army to many successes, including the Second Route of the Jiangnan Camp, capturing Suzhou and Hangzhou, and even attacking Shanghai.
This is not to say that the Taiping didn't suffer from leadership problems, but I wouldn't go so far as to call it a "complete breakdown." Certainly the Tianjing Incident was devastating for morale, but the fact that the Taiping survived for another eight years and continued to enjoy battlefield success should indicate that its leadership was still functioning in an effective manner. Far more important reasons for their failure are as follows:
Failure to enlist Western aid:
Hong Xiuquan was a Chinese nationalist who was against opium and wanted to repeal the Unequal Treaties imposed on China by foreign nations. This, and the fact that his version of Christianity clashed with both Catholicism and Protestantism, led the Western governments to back the Qing so that their economic privileges in China would be protected. Western forces proved instrumental in the defense of Shanghai, Western officers trained Chinese troops and provided Chinese armies with Western weapons.
Hong Rengan considered Western support for the Qing to be the main reason why the Taiping Rebellion failed.
[我朝禍害之源，即洋人助妖之事。]
"The source of our dynasty's calamity is that the foreigners aided the devils [Qing]."
[如洋人不助敵軍，則吾人斷可長久支持。]
"Had the foreigners not aided the enemy army, then we [could have] continued for a long time."
Failure to elicit public support:
By public support I mean members of the gentry, whose participation in the war on behalf of the Qing was instrumental to the Taiping's defeat. Zeng Guofan, Li Hongzhang, and Zuo Zongtang were all leading members of the gentry who were primarily responsible for defeating the Taiping. In his "Order to Suppress the Rebels from Yue," Zeng Guofan wrote the following in regards to the attitude of the Taiping towards traditional Chinese religion:
[自古生有功德，沒則為神，王道治明，神道治幽，雖亂臣賊子，窮凶極丑，亦往往敬畏神祗。李自成至曲阜不犯聖廟，張獻忠至梓潼亦祭文昌。粵匪焚郴州之學宮，毀宣聖之木主，十哲兩廡，狼藉滿地。嗣是所過郡縣，先毀廟宇，即忠臣義士如關帝、岳王之凜凜，亦皆污其宮室，殘其身首。以至佛寺道院，城隍社壇，無廟不焚，無像不滅。斯又鬼神所共憤怒，欲一雪此憾於冥冥之中者也。]
"Since ancient times, those with virtue and achievements while living become gods after death. The Way of the King governs the light, the Way of the Spirits governs the dark. Even though [they were all] rebels and traitors, savage and disregarding proper ceremonies, [they] also frequently revere and honor the spirits. [When] Li Zicheng arrived in Qufu he did not violate the shrine of Confucius, [when] Zhang Xianzhong arrived in Zitong he also made sacrifices to Wenchang. The rebels from Yue burned Chenzhou's school and temple and destroyed the wooden tablet of Confucius, the side halls [worshipping] the ten disciples [of Confucius] are in complete disarray. Every prefecture and county [the rebels] pass, they will first destroy the temples. Even [the temples of] the loyal ministers and righteous persons such as the revered Guan Yu and Yue Fei are also defiled, their statues destroyed. As for Buddhist temples, Daoist shrines, and altars to Chenghuang, there is no temple they would not burn, there is no statue they would not destroyed. These spirits are all angry, and they wish to [cover] these regrets with snow in the dark."
The Taiping represented a direct attack to traditional Chinese culture, and this caused them to lose public support. You can almost say that the rebellion was doomed to fail from the beginning. It was just far too radical.
One also has to consider that ever since it captured Nanjing, the Taiping was pretty much fighting a defensive war (with the exception of their failed Northern Expedition to take Beijing). Almost all of their strategic decisions was made on the basis of relieving pressure on their capital, which was under siege for the great majority of the war. They were not in a good position to win.