Part 1 of 2:

My day has come!

Explanation for those who don’t know who the Shinsengumi are. In the early 1860s, the Imperial city of Kyoto descended into complete civil chaos with ronin streaming into the city, patriotic assassinations (read thuggish murders), robbery/shakedown of the citizens, and the threat of civil war looming. Having completely lost control of the city, and trying to secure it for the shogun’s 1863 trip to Kyoto from his home in Edo, some bright minds in the shogunate came up with the idea of fighting fire with fire: why not recruit ronin from the Edo area to help secure Kyoto?

The immediate problem was that some of those same bright minds, particularly a man named Kiyokawa Hachiro, also thought the shogunate should join up with the patriot ronin in Kyoto and begin a war of expulsion against the Westerners. At public meetings around Edo in late 1862, with the full support of the shogunate, Kiyokawa and other recruiters raised a force of about 250 men. The force consisted of samurai who’d fled their domains and gone ronin, peasants and townsfolk with ambitions to be samurai, and even some guys let out of jail for the occasion.

The subsequent history of this force was every bit the disaster you can imagine. Conrad Totman wrote a brief account of their early days. Having arrived in Kyoto in early 1863,

it provided a sometimes useful police force with extensive connections to the network of activist samurai. Its men were not always trustworthy, were guilty of some acts of savage violence, and quarreled among themselves and with other shishi. (p. 49, The Collapse of the Tokugawa Bakufu, 1862-1868 by Conrad Totman)

Very soon after they arrived at Kyoto, this bunch turned around and went back to Edo, apparently encouraged by the local authorities in that direction. Kiyokawa’s new plan was to go fight the barbarians in the Edo area. He and his force continued to spread havoc wherever they went, the shogunate eventually had Kiyokawa assassinated, and many of his men arrested. The remainder were folded into Shonai domain’s forces in Edo. The venture was a disaster and the whole sorry story is well-documented in Totman’s massive volume.

However, thirteen members of that ronin force decided not to go back to Edo. The Roshigumi (ronin group) rented two houses in a small village, Mibu, on the outskirts of Kyoto, and started looking for some official patronage. The daimyo of Aizu Domain, Matsudaira Katamori, had recently been appointed to a new position: Kyoto Protector, charged with bringing order back to the city. Aizu officials took these ronin under their authority, and gave their unit an official name: Shinsengumi (new selected group). In modern terms, we might call the Shinsengumi a paramilitary group. They were under the ultimate authority of Aizu, but not incorporated into Aizu's organization directly.

As your question mentions, these men became legends. Polls of the most popular Japanese historical figures of all time regularly have Shinsengumi members in the top ten. So, now we get back to your original question.

How well were they known in their own lifetime, and how much of their current fame is owed to 20th century fiction? What did the Shinsengumi actually achieve?

In their own time in Kyoto, of course, they’d be very well known. The Shinsengumi was actively patrolling the city, they were big spenders in the pleasure quarters, their actions were often very dramatic. One well-known incident involved early Shinsengumi leader Serisawa Kamo firing a cannon on a shop he claimed was supporting the anti-Bakufu ronin, another involved a street fight in Osaka between Shinsengumi members and sumo wrestlers.

And then there was the Ikedaya Affair, which is the Shinsengumi’s big claim to importance in the larger history of the period.

The Ikedaya Affair, on July 8, 1864 is an iconic moment in the Bakumatsu period. Acting on the information of its spies, the Shinsengumi took prisoner a shop owner named Furutaka Shuntaro, who was affiliated with anti-shogunate activists in Kyoto. Under interrogation (Furutaka was certainly tortured but any details on the internet come from fiction), Furutaka confessed to being part of a plan, in coalition with Choshu domain samurai, to set Kyoto on fire, and use the distraction to kidnap the Emperor. From his confession and other information, the Shinsengumi concluded that the conspirators would be meeting late that night at one of two inns in Kyoto. They notified Aizu domain of the operation, and when Aizu delayed sending back-up, the Shinsengumi split their force of thirty men into two groups to go to the two inns.

The smaller group of eleven samurai, led by Shinsengumi chief Kondo Isami, raided the inn, the Ikedaya, where the meeting was taking place. The conspirators fought back. In the fighting that followed, three Shinsengumi members were killed, and seven of the activists. Reinforcements from the other Shinsengumi group arrived after a while, and the Aizu and Kuwana domain troops surrounded the inn. Twenty-three of the activists were then taken prisoner. Some of the dead and prisoners were samurai of Choshu domain.

This was, without a doubt, the crowning moment of glory for the Shinsengumi. They received official thanks, reward money, and a more respected role in Kyoto law enforcement. The Ikedaya Affair also had a strong effect on the Choshu militia troops who were already raring to fight Aizu and Satsuma (at that time, the two domains were allied) in Kyoto. The Ikedaya Affair wasn’t responsible for putting Choshu on course to invade Kyoto, but it definitely helped take the brakes off that force. On July 19, Choshu forces entered the city with the intention of taking control of the Emperor. In the ensuing battle, known as the Kinmon no Hen or Hamaguri Rebellion, Choshu was defeated by Aizu and Satsuma forces, with the Shinsengumi fighting alongside Aizu’s troops. To cover their retreat, the Choshu troops set fire to the city, so ironically, the very scenario the Shinsengumi had set out to avoid, came to pass.

The big question ever since then, of course, has been whether the conspirators did in fact plan to burn Kyoto. My suspicion is that it was on the table, a strategy that could be used in case of a battle, as indeed actually happened, not something the conspirators were set on doing.

I can’t say how far these events were known outside Kyoto. There were popular news notices - one page printed bulletins called “kawaraban” - which were very commonly read in Edo, and it’s possible stories like the Ikedaya Affair made it to the public via that avenue. Kondo Isami’s execution, years later, definitely was spread by kawaraban.

In the hometowns of the core Shinsengumi members, villages just outside of Edo, the Shinsengumi were definitely super stars. The Shinsengumi did go successfully recruiting after the Kinmon no Hen in Edo, so that would also have meant a fairly high profile.

It’s not that the Shinsengumi did nothing after the Ikedaya Affair; they continued to have exciting careers, and then in the Boshin war, their members either as a group, or on their own, were very active. But the Ikedaya Affair was the moment where the groups’ actions were most directly important to the "big events".

If the Ikedaya Affair was the Shinsengumi’s greatest moment, they probably owe their undying fame to their worst moments: the tragic fates that most of the members suffered: Kondo Isami’s execution by the Tosando army after he had surrendered, Harada Sanosuke’s death fighting with the Shogitai at Ueno, the great swordsman Okita Souji dying of tuberculosis unable to join the war, Hijikata Toshizo carrying on his battle to the wilds of Hokkaido, Saito Hajime refusing to leave Aizu behind and surviving among its prisoners of war. As Ivan Morris would put it, Japan loves “the nobility of failure.”

So, for instance, while I can’t say when the Shinsengumi became a household name, they were certainly famous before the 20th century. Portrait prints of heroes on the shogunate side of the war were extremely popular in Tokyo (as Edo was renamed). Tsukioka Yoshitoshi put out this engraving of Kondo Isami in 1874, six years after Kondo’s execution, evidently assuming that the people of Tokyo would know whom he was, and would want to buy it.

As per Ernest Satow's version of events, Satow’s book is great, but he wasn’t up on everything going on around him, particularly on the shogunate side. He was never sympathetic to that side, did not grasp the intricacies of shogunate internal politics, and was of course, himself a major player in Bakumatsu politics, a boon to the side of the South-west domains. For events he wasn’t involved in himself, he obviously relies on a Choshu narrative.

Interestingly, he never even mentions Sakamoto Ryoma in his book, despite having apparently taken at least one voyage on board the same ship as Sakamoto. You couldn’t conclude from his memoirs that Sakamoto thus wasn’t important to Bakumatsu events, it’s just Satow wasn’t knowledgeable about his story. The same goes for the Shinsengumi, although, I would agree that they were not extremely important.

Full-blast Shinsengumi-mania took a while to form. At this point the Shinsengumi are some of the biggest heroes of the Bakumatsu, crowding out many of the other potential shogunate heroes. This reputation does owe a debt to fictionalizations, such as Shimosawa Kan’s novels, and the Kurama Tengu stories and films, in which the Shinsengumi are the bad guys, but the important increasingly noble adversaries of the main character.

I’ve tried to keep this answer to the point, without going into a full account of the Shinsengumi and their times, so feel free to ask for elaboration, or about any particular aspect or member of the Shinsengumi.