Romanticism, through and through. That's what that is- the romanticized idea of an individual master able to conquer all that comes before them. Sometimes they're a just hero, sometimes they're anti-hero, often in pop culture they're a reluctant hero. It plays to societal desires, especially in modern times when Toshiro Mifune and other dominated samurai films. Who doesn't like a badass like Mifune??

But this ideal wasn't confined to purely modern times. Since the Edo era, there have been stories and romanticized ideals around samurai and ninja. Tatsukawa Bunko books emerged in the early 1900's were kodan (講談) or kodan-bon (講談本) (lit. "story-telling" and "story-telling books"), marketed towards younger audiences, meant to capture imagination. This wasn't much different from monogatari (物語, "tale, story, legend" - lit. "word thing") that had been around for centuries. They're like epic novels, Lord of the Ring equivalents, which prosey and fantastical tellings that may or may not be based on actual happenings. The Heike Monogatari (平家物語) may be the most famous of these. Compiled in the early 14th century, it focuses on the Genpei (源平) war, between the Minamoto (Gen, 源) and Taira (Hei/Pei, 平) families that ultimately resulted in establishing the Kamakura shogunate under Minamoto Yoritomo. However, in the story is the famous Minamoto Yoshitsune, a charismatic, brave warrior and military genius, who learned military secrets from tengu on Mount Kurama and was accused of treachery by Yoritomo. Yoshitsune has been romanticed ever since.

Another example I like to use is the Hagakure (葉隱), from which what I call bushido (武士道) populism starts to emerge. Bushido is popular in modern times; the whole "samurai honor" and "the way of the warrior is death" mentality is so far from the truth of the time. When the Hagakure was released in the early 1700's, it was viewed as a nostalgic reflection upon the era of the warrior class and cultivating one's mind through budo. It was a north wind vs south wind mentality; you had the elders who constantly try not to forget war and then the discord of soldiers and others trying to live in a peaceful society without war. The Hakagure at the time of its release was frown upon by those of peace as the "ideological pasttime of a thinker over-familiar with tranquility and peace" and "no more than empty theory, unaccompanied by practice" as the author had no training or wartime experience, which supports the "mystique" aspect to warfare and the allowance of swords at the time. Real warriors of the day saw it as trash, commoners thought it was pretty spiffy. It was eventually forgotten until the 20th century when modern military used it as a sort of "inspiration" to troops based on, again, romanticed ideas of what samurai were like.

Real samurai stories are far, far, far less glamorous. There is a lot of murder, lots of rape, tons of abuse of power, scheming out the wazoo, etc. But that doesn't get focused on because that doesn't fit the narrative. Similar to situations in Europe with the "knight in shining armor" glasses that people may wear.

All that being said, your question opens up a path to discuss something interesting: military concepts of pre-modern Japan. You make reference to Zulus and Romans with their military formations; those are state-sponsored (or, in Shaka's case, staet allowance of systematizing training). This is importatnt because there was no state military in pre-modern Japan. In the Heian era (794-1185), Japan was not guarded and policed by "imperial swords," but by hired swords- the private military resources of a rising order of professional "mercanaries." As Dr. Karl Friday puts it, this order had come to monopolize the application of arms in Japan by the early tenthy century, the emperor and his court having cast aside the original military institutions of their state a century before.

And it's true; in 792, the court abolished the provincial garrisons that had made up most of the military forces created by the ritsuryo (律令) codes. Although the order did not include the provinces of Mutsu, Dewa, Sado, and those under the jurisdiction of the Dazaifu (an incredibly important border gateway in Kyushu), the perception is/was that this effectively spelled the end of generalized peasant conscription in most of the country. What that view fails to recognize, however, is that the government's edict in 792 was just one step in a continuing series of reforms and modifications directed toward a rationalization of the state's armed forces. From Dr. Friday-

the edict of 792 created an institutional vacuum in the military system in all but the frontier provinces. The court had recognized the weaknesses of the earlier system and moved to correct them. It had identified its best source of military power, but had not yet decided upon an organizational framework within which to utilize this group. As a result, there was more than a century and a half of groping and experimenting on the part of the central government. Various ideas were tried, modified or abandoned until the court achieved a workable system during the middle 900s. There were two problems to be solved: by what means were troops to be called up, and by what sort of officerships were they to be led? The development of the system, as it struggled with both problems, reflected the emergence of the warrior class and the efforts of the court to keep pace with this evolution.

From the inception of the ritsuryo, it's possible to identify a pattern of increasing reliance on the martial skills of the rural elite (gunji 郡司, lit. "district officer") and upper-tier peasant classes, and lessening use of the ordinary peasantry in both the capital and the provinces. This trend is seen most clearly in the evolution of the various military/guard offices in the capital. The ritsuryo codes established five such organizations, collectively known as the Goefu 五衛府 (five guards): the Emonfu 衛門府, Left and Right Ejifu 衛士府,and Left and Right Hyoefu 兵衛府. Troops for the Hyoiefu were drawn from among the lower central and provincial nobility; the Ejifu and Emonfu were staffed by troops (called eji 衛士) taken from the provincial brigades and serving in rotation at court. In terms of numbers, these peasant guards were by far the largest force, but they quickly came to be viewed as unreliable. Beginning in the early 700s, the court began to compensate for the inadequacy of the eji by shifting some of their functions to the Hyoefu and others to newly created units. By 811 the Goefu had been replaced by the Rokuefu 六衛府, or Six Guards, and the involvement of ordinary peasant conscripts was minimal. The Court fleeting with kebiishi (検非違使, a type of police/judicial chief), oryoshi (押領使, ), tsuibushi (追捕使), and more.

Peasant conscripts proved to be no better soldiers in the provinces. An edict issued by the Daijoukan (大政官, Council of State) in 780 complained of the enervated condition of provincial garrisons. Most, it noted, were not even trained, but simply issued weapons and then used as private labor forces by provincial governors and military officers. Accordingly, the numbers of troops assigned to the garrisons were reduced, and provincial officials were directed to focus their conscription effort on those among the upper tier of peasantry who were well versed in the skills of "the horse and bow" for calvary had emerged as dominant in Japanese warfare by the 7th century. In fact, it was paramount to Emperor Temnu's victory in the Jinshin War in 672. The imperial military at the time was meant to serve two purposes: defend from continental invasion and thwart challenges to the imperial court/court edicts by regional nobility. After all, Japan was still a feudal country with genuine stateship only just beginning to solidify. That's why in 762, the government had ordered the formation of special units in Ise, Mino, Echizen, and Omi- the provinces guarding the major approaches to the Heian capital- to be composed of expert mounted archers drawn from the scions of provincial aristocrats and wealthy peasants.

I know this is lengthy, but we're getting to the point in regards to public/private military, formations, and the like. Bear with me! (cont)