The Shōgitai is perhaps a good demonstration of the personal interactions that caused what would perhaps be objectively speaking, like pointed out in the question, illogical and unnecessary destruction.

So, the first point to get out of the way is that Tokugawa Yoshinobu was not seen as an incompetent coward. Contemporary records from both sides depicted him as charismatic and efficient, frightfully so to the leaders of Satsuma and Chōshū.

This brings us to Shibusawa Seiichirō. Seiichirō and his younger cousin Tokudaiyū were born to peasants in what is now Saitama Prefecture, named Kisaku and Eiichi respectively. Like many peasants of the time period they were educated and caught up in the wave of anti-foreignism. But when they applied to the Hitotsubashi clan in 1864, wishing to find a way to contribute (after running into dead ends on their own), Yoshinobu found a way to put the cousins to work. At the time Yoshinobu as the Hitotsubashi daimyō was also Shōgun Regent and Protector of the Imperial Palace. However, when Tokugawa Yoshimune formed the Hitotsubashi clan for his younger son, along with Tayasu and Shimizu, the clan was only supposed provide successor to the Shōgun in case Yoshimune's descendents on the Shōgunate family dies out. Hitostsubashi was merely 100k koku. Not insignificant, but far behind the big players. 100k koku would not even put him in the top 30 clans. In comparison, Satsuma (then an ally) was 728.7k. Chōshū was 369.4k. And these are official numbers, with significant left overs from the Edo Bakufu's founding and masks population and economic growth that took place during the era. Tax census taken in 1869-70 actually put Satsuma at 869.6k and Chōshū at 988k (Chōshū is a special case as in addition to growth the official numbers was also artificially kept low due to politics). The point here is that if it came down to a war mobilizing men, money, and resources, Yoshinobu was far out of his league. Yoshinobu had wished he could rely on his birth family of the Mito clan, but Mito was fighting a rebellion. Even worse, the Hitotsubashi was not a consolidated domain that came out of the wars of the Sengoku, but various pieces of fiefs all over Japan that only sums up to 100k koku when combined on paper. In an attempt to at least create a semblance of his own military force, Yoshinobu ordered Seiichirō to travel the Hitotsubashi realms and convince peasants to volunteer to join his army. Of course, Yoshinobu was far from the only one to realize the potential to form armed forces from the general population. However, this is an example that he was not blinded by old thinking and was very willing to use what resources he had at his disposal and to promote based on merit. Seiichirō for his accomplishment at gathering troops was promoted, and when Yoshinobu became Shōgun in 1867 he made Seiichirō his scribe. Imagine a peasant whose family was in the trade of making dyes and growing silkworms becoming the Shōgun's scribe in three years. As an aside, at the time, Tokudaiyū was ordered to accompany Yoshinobu's brother Akitake as part of the Bakufu's diplomatic mission to France and to participate in the International Exposition of 1867 in Paris.

The next point that could be seen from the above is personal loyalty. Seiichirō owed everything to Yoshinobu. And of course in Seiichirō's eyes Yoshinobu was not an indecisive coward, not least because Seiichirō must have heard of Yoshinobu bravely leading the defense of the imperial palace against Chōshū attackers in 1864. Seiichirō also had absolutely no place outside of the Bakufu except a little peasant village. To Seiichirō the blame laid not with Yoshinobu, but with Satsuma that had betrayed the bakufu to Chōshū, and of course Chōshū. Seiichirō himself, like many young men of action at the time, was also fairly hotheaded. In 1863 when he was 25, he and his cousins along with a group of young men of his village and surrounding area actually planned to storm a nearby castle and then attack Yokohama and burn it to the ground (it was never carried out). Now in 1868 he like many others advocated that they continued to fight. It was of course Yoshinobu's own idea to swear allegiance to the new government and place himself under voluntary confinement, something Yoshinobu pushed through over many in the inner bakufu who wished to fight. But would Seiichirō have known that, or would he have blamed people who pushed for peace? Whatever the case, when Yoshinobu voluntarily left Edo and confined himself to the temple at Ueno on March 5, Seiichirō and many others gathered there ostensibly to protect Yoshinobu, but they were also there to try to convince him to continue the fight. More and more men gathered there, and they elected Seiichirō as their leader, forming the Shōgitai with an initial 300 men. Yoshinobu certainly did not want their "protection" and rather seemed to have feared the Shōgitai, and the group played a part in his request to be transferred to Mito for his confinement.

And here enters another character, the 21-years-old Prince of Rinnōji, later Prince Kitashirakawa Yoshihisa. the Prince of Rinnōji was an imperial prince the Edo Bakufu kept nearby at Ueno. In theory this was done in case there ever was a war between the Bakufu and the Imperial Court, the Prince of Rinnōji would be declared the righteous emperor and give the Bakufu the imperial authority to keep fighting. Well this Prince was asked by Yoshinobu to go to Sunpu to let the Imperial Army know of his surrender and ask them to call off their expedition. On March 30, the Prince met the commanders of the Imperial Army in Sunpu, who accepted Yoshinobu's personal surrender. But on April 4 the Imperial Army told him while Yoshinobu's surrender was accepted, as could be seen by their attitude towards Aizu, the Imperial commanders were determined to continue the fight and squash out all possible resistance. Had they taken a softer stance perhaps there would not have been so much fighting, but whether out of a sense of revenge or to solidify their place in the future government, getting rid of resistance was their foremost priority. The Prince seemed to have been incredibly angered by the entire experience and returned to Ueno and began to call for active fighting. In late June as the situation at Ueno was headed for open fighting, Katsu Kaishū, the man who negotiated Edo Castle's surrender, recorded in his diary that he heard the Prince was even going to take up the Imperial Banner himself, ie, declare himself the rightful emperor. Recall here that he was supposed to be Yoshinobu's diplomat for peace.