The perception of Samurai being exclusively mounted archers is just as flawed as the one where they’re katana-wielding warriors marching on foot in the heat of every battle. In reality, the samurai were more an economic/political caste, whose role as warriors came and went over the years.
The mounted archer “samurai” was most predominant in the Heian period, (790-1190, give or take 5 years.) During this time, samurai were indeed mostly archers who rode into battle on horseback in flashy armor, though the horses and armor were mostly there to flaunt wealth. Samurai occupied a role one step below that of the Daimyo, or local liege lord. (Daimyo were in turn one step below the Emperor, then later the Shogun, though especially during this time period their agency within their lands was far greater.) Thus, samurai were mostly either court bodyguards or mid-low level nobles (or both), who only occasionally had to march out and disperse peasant revolts or small invasions. During these revolts, they were for the most part greatly outnumbered and up against men with spears and little training or morale. The Katana just isn’t useful during these engagement, so setting up 100 men on a hill and pelting the crowd with arrows (all while looking impressive and rich) usually did the trick. When it came to disputes over territory or what have you between local lords, usually champion samurai were sent out to fight it out, one on one.
This strategy famously fails the samurai of Japan during the invasion of the Mongol Empire in the tail end of the 13th century, and thus the kamakura period (which established feudalism and the daimyo + samurai cast as far more important) ends as one shogunate (the Ashikaga) replace another, (the minamoto.) at this point, samurai are truly samurai, owning small fiefdoms of their own sometimes and even endeavoring at times to spread their own political power.
The warring states period 200+ years later greatly expands the role of the samurai though, who now often take up positions as captains or generals within larger armies. Though the vast majority of soldiers during the time are still just peasants with spears, Samurai increasingly join the fray wielding pole arms, bows, and katana, and as a last resort, short swords known as wakizashi. Still, even during this time, we don’t see many Katana wielding monster samurai.
The warring states period ends with the unification of Japan, and a great period of peace that lasts nearly 250 years. During this new Edo period, samurai mostly lose their day jobs as warriors. They take to academia and courtly business, developing mathematic and philosophical systems, as well as theorizing on science and politics. They maintain their positions, to some extent their power, but with almost no battles to fight over 250 years... the only swordsman samurai still present were third or fourth generation students at swords dojo’s, more hobbyists than real warriors (even though some of them were pretty damned skilled.)
To keep an already long answer from dragging any further: Meiji restoration comes. Half of Japan wants to modernize, half of Japan wants a return to the status quo. Samurai lose their positions early on in Meiji, (were up to the 1870s-1880s by now) and the caste system as a whole breaks down. THIS is when the master swordsman samurai myth emerges and takes shape in a way we might recognize it today. Those who opposed modernization and the end of a long isolationist period harkened back to the past, or a rather fictionalized version of it. Samurai were great warriors now, not politicians or landlords. They fought noble battles with swords, layered 7-8 times over on the forge to be able to pierce STEEL. Thus, a legendary class of warrior roots itself into the zeitgeist of a tense period in Japanese history, and the role, bravery, nobility, and legend of the samurai becomes greatly over exaggerated.
TLDR: Samurai were never entirely mounted archers, nor entirely sword-swinging masters of combat. Sometimes they were a private police force for daimyo, sometimes they owned small bits of land and fought on the side, and sometimes they were a caste of scholars and mathematicians. The image we have of samurai now is greatly influenced by pop culture, which was in turn greatly influenced by the propagandized version of Samurai heralded by the last holdouts of the Edo period, men challenged by modernization and looking for a return to the glory of the past.
Source: graduate student of East Asian studies focused on the theory of nation state sovereignty during the Meiji restoration.
Edit: a word
While reading up on my material for this question,I was actually wondering how to answer this question or even if I could as it seemed like a question without end. The popularity of the katana was something honed from many periods (although there are a few obvious culprits like the Edo period and the Meiji restoration) and is very likely not due to any specific reason in general but a whole plethora of reasons. I then wondered if I could somehow narrow it down,maybe list a few reasons and give some resources on the other reasons.
EDIT : while typing out this answer,I discovered that a partial answer had already been posted....Well,feeling pretty darn dumb right about now.Nevertheless,I hope those that do scroll all the way down here enjoy the read!
I'd feel a little bad,however, if I din't give it my best shot,so let's list ALL of them.As the story is a little long,I welcome any and all corrections as well as other possible factors I might've missed. So,without further adieu,
TLDR's Below Pls I swear it's a good story
The Katana was never replaced by horse archery and/or spears
Contrary to popular belief,the katana wasn't actually disused in the early Heian; Swords have a history as old as Japan itself albeit under different names.They might not have been common-issue for every levy, but nearly anyone who could afford one would have one and the metalworking shows. Single-edged blades of hand-forged,high-carbon steel appear around 400AD , were called Chokuto and looked like this . They were forged in the style of Mainland Asia, specifically in the Chinese mold. While made of steel,unlike the bronze swords of even earlier eras,they do not hold up to modern standards of rigidity and lack the strength and structure required for warfare and hence have been speculated to be for ceremonial intent.In the ensuing centuries,rivalries between noble clans erupted into occasional warfare and this is where the image of the horse skirmisher appears,samurai armed with bow and arrow, to fight on the battlefield. The tachi shown here is a by-product of the evolving needs of warfare and a mainstay of how Heian period (794 to 1185) swords looked like. (You'll note the slowly curving shape taking the popular image of the katana)
The Kamakura period (1185–1333) saw the tachi's evolution,primarily due to the Mongol Invasions. The delicate design that saw the tachi's beloved use throughout all of the Heian period saw damaged swords and chipped steel when dealing with the heavily armored Mongolians. Hence it evolved,again,developing hardened steel sheaths wrapped around soft ductile cores,becoming two handed. Knives known as tanto became part of the samurai arsenal for use in close combat and longer,one-handed swords known as uchigatana or katana were introduced. A Kamakura tachi looked something like this .
Broadly speaking,all Japanese swords made before 1596 are called koto (early swords) and those after ;Shinto (new swords). The reason for this seemingly arbitrary divide is simple : Shinto swords were leagues better than Kamakura period swords. Japan had improved both in its metal refining and in its forging techniques since the Kamakura period and swords were often bright,polished,adorned,and overwhelmingly katana not tachi. By the beginning of the Edo Period,nearly every sword that adorned the hip of a samurai was a katana,often accompanied by a wakizashi short sword,known as a daishō,
Change hit the samurai class of the Edo Period like a truck (This deserves a whole other question on its own haha) and saw the general impoverishment of the samurai class who frequently found themselves in debt and unable to afford expensive swords. In a similar vein, blacksmiths found themselves lacking more and more customers and began forging swords of lower quality. Unauthorized forgeries of swords made by famous swordsmiths such as Suishinshi Masahide were common and his forged signature was placed on the tang of many forgeries. Swords became commonplace,even among the non-samurai class and anyone who could afford one,could theoretically get one,even if they couldn't carry one in public.Moreover, even near the end of the Sengoku Jidai,it was obvious that the matchlock rifles had a found a permanent place in Japan's military. The art of swordsmithing, while still revered,was not necessarily the sole profession a blacksmith could become. He could also become a gunsmith and those were constantly in demand,further contributing to the decline and rarity of the katana. Despite the general drop in quality of swords,most samurai still carried their cheap steel for one reason ; they were samurai and to carry swords was a symbol of their status as demonstrated by u/wotan_weevil here . Throughout the Edo period,the Katana sword,despite its sharp drop in usage,utility and quality saw its continued existence as a partner in nearly all aspects of affluent Japanese society. It was an old and valued partner that had accompanied Japanese society since its inception and never left,firmly imprinting itself into the psyche. Sure the Japanese bushi and ashigaru might have used bows and arrows but the sword was not thrown aside nor was it simply abandoned; it was just used less and the sword in everyone's minds after 200 years of the same "model" was the Katana.
The Black Ships arrive and the Katana becomes cool(er)
The Meiji Restoration saw the dismantling of the samurai class,the revoking of their government sponsored stipend and the removal of sword carrying privileges for the samurai even saw a disgruntled faction rebel in 1877. The rapid industrialization of Japan followed soon after,closely followed by its imperial ambitions,some of which regarding Korea I've highlighted here . While initially,the disenfranchised samurai, such as Ōki Takatō might've regarded their Edo privileges as a shameful loss (He sympathized with the Satsuma rebellion in 1877 despite being part of the imperial government),perhaps the most important members in creating katana fondness were the Meiji thinkers instead. Mori Arinori,the founder of the Japanese education system, stressed the necessity of a japonicized standardized education,one with focus on Giri (duty) and loyalty. State Shinto and the modern conceptualization of Bushido described here and here only accentuated the fondness for the "strong" days of Edo Japan,when Japan was free to be Japan,when it could embrace its uniqueness and be free from western influence. Fukuzawa Yukichi ,the famous author and critic was outspoken on Japan's perceived leanings to the West. It could learn from them,yes,but not abandon its roots,paving the way for the final phase of Katana nostalgia.
Greetings. Not to discourage further responses, but do check this older answer while you wait :
How did the sword become the iconic "medieval weapon" when other weapons of the time were either stronger aganist plate or had more reach with comparable damage? by u/bigbluepanda
Hope this helps.
It's interesting to consider the "katana culture" of today when looking back at the history of warfare in Japan- ie, bushi calvary with bows and ashigaru with spears and the like. Sword has always been important to Japan's history, going all the way back to the emergence of the state of Wa and Yamato-era governments. Like all weapons, it was based on a particular need- killing in a closer range. We see their usage regularly in Japanese mythology and such instruments were regularly presented as rewards and gifts by regional leaders to their followers. In the Kamakura and Muromachi we already see phrases like tachi uchi 太刀打ち (lit sword striking) as a general method of referring to confrontation. The spiritual significance of the last 2,000 years is not something to be ignored.
From a warfare perspective, even though calvary was the main approach in large scale battle, swords (katana, tachi, nodachi, tsurugi, warabite katana, etc) had their place as backups- much like a sidearm pistol for a soldier today. The evolution of the tachi from straight blade to the varying degrees of curve can be seen with the evolution of armor styles and materials. That in turn changed the way swords could be carried and drawn. This was important when swords needed to be used from horseback- after arrows and long weapons like yari and naginata failed to be of use. In a scholarly chicken-egg argument, some scholars felt sword curves emerged as a result of the needs by calvary; more modern scholars such as Dr. Karl Friday point to the change in curvature as an engineering solution with a calvary-enhancing by-product than a need. After all, Friday points out calvary usage had been trained as part of the military since the 700's when straight blades were the only thing to exist.
The point being made is that swords have always been around, even with calvary, and their evolution in engineering had a role in how they were worn/used by said calvary. But what about the crux of the question, what about the "sword culture" ?
As alluded to at the beginning, the sword culture as sword awareness, knowledge, and usage always existed. Swords began as a weapon of the elite and nobility, which is why you'll see paintings of Nara era figures with a straight sword as part of their attire. When the Ministry of Military Affairs (hyobusho 兵部省) came into existence as part of the ritsuryo codes in the 7th and 8th centuries, training include horse, bow, spear, and sword. After the codes were abandoned and reformed and the military became privatized (much larger separate discussion), the need for swords and sword training continued. We know that as early as the 9th and 10th centuries that "skill" with a sword was recognized, and depending on the history of ryuha (martial art schools) in Japan you subscribe to, teachings of things like the Kyo Hachi Ryu and more began to take shape. Those schools would be considered sogo bujutsu (complete bujutsu, ie focusing on all weapons).
Others here have already described the place of swords in combat. Large scale battles they aren't favored and shouldn't be favored; Kamiizumi Ise no Kami, the founder of Shinkage Ryu (arguably the most famous "sword school" in Japan) was a master of spear. Sword use in private squabbles, assassinations, and grapplying situations has always existed. Thus when we think of "sword culture" and the "favor of swords," we need to look to the Edo era to really get a sense of when the "sword" become more "mystical."
In 1588 the katanagari (刀狩 sword hunt) edict from Hideyoshi comes out. This reform sought to solidify the class structure by prohibiting the possession of swords and guns by all those who would be the "noble classes." Claiming the possession of weapons by peasants "makes difficult the collection of taxes and tends to foment uprisings," the mandate prohibited farmers from possessing long or short swords, bows, spears, muskets, or any other form of weapon. Here then the sword moves from a common place amongst the lives of every day folk to one of priviledge. It has special meaning, one of status, one that shows "betterment" in the eyes of some... and "not so betterment" in the eyes of those who would be preyed upon by those classes (daimyo, samurai, etc).
Interestingly enough, the "sword culture" as "favor of swords" also appears in the form of an increase in kenjutsu ryuha in the Edo period, ie martial art schools that focused only on swordsmanship. These differed from the older, sogo bujutsu like Katori Shinto Ryu, Kashima Shin Ryu, Shinkage Ryu, Nen Ryu, Chujo Ryu, etc. The emergence of these schools is a result of some warriors needing to make money after the war (ronin or otherwise), supposed "tough guys" looking to cash in and teach their "way of the sword," and others who simply wanted to pass their traditions on. In some cases, the soke (headmaster, lit. head of the family) was also employed in a government position and the schools were meant as ways of teaching state and/or provincial employees.
But regardless of the origins, the general glamor surrounding skill in combat that has existed since civilization began coupled with schools that focus on being the superior with a sword, coupled with the sword being a status symbol all contribute to the "sword culture" you refer to. Consider as well the Hagakure written in the early 1700's as a nostalgic reflection upon the era of the warrior class and cultivating one's mind through budo. It was a north wind vs south wind mentality; you had the elders who constantly try not to forget war and then the discord of soldiers and others trying to live in a peaceful society without war. The Hakagure at the time of its release was frown upon by those of peace as the "ideological pasttime of a thinker over-familiar with tranquility and peace" and "no more than empty theory, unaccompanied by practice" as the author had no training or wartime experience, which supports the "mystique" aspect to warfare and the allowance of swords at the time. Real warriors of the day saw it as trash, commoners thought it was pretty spiffy.
/u/MRBEASTLY321 pointed out the further evolution of "sword culture" in the Meiji era in the tradition vs modernization dillemma being faced. That did add to the mystique and reverence of the samurai, while also leading to a morph in image. The samurai generally perceived today was definitely not the samurai of history. Pop culture tends to prop up that which "appears cool" regardless of whether it was cool or not. It's like the perception of ninja wearing all black and hoods when history very much shows that not to be the case.
I'll end this by saying the modern "sword culture" as seen in anime, movies, video games, etc. likely exists, in my opinion, as a result of the status perception throughout history coupled with the "cool" factor that sword skill brings with it. "Easy" fights with bows and arrows don't make for good entertainment unless you're John Wick. Even long weapons like spears don't find wide use because of how difficult it is to render a spear vs spear or sword vs spear type fight. It's been successful in some cases, but the regular Joe understand skill with sword, understands the closeness required, understands the risk and, thus, props it up and creates the "sword culture" you refer to.
Sorry for this being long-winded and jumping around. I thought it important to cover some history around sword use and the evolution of it's use to point out that "sword culture" has always existed and simply changed with the times.
Sources-
There have been some interesting answers about historical Japan, but on the other side of the issue, I have a narrower sub-question about the evolution of cultural depictions. Kurosawa's protagonists were often sword-toting samurai; how much was this based on his reading of authentic Japanese history vs. adapting his own version of the Western film with a gun-toting cowboy? Was Kurosawa's vision of historical Japan a major influence on later depictions, or where else did 20th-century artists get their ideas about samurai?
