It is true that there were very few militaristic threats that required the samurai to take up their swords during the Edo period. The warrior aristocrat class that the samurai belonged to had brought with it stability in it's founding; some of these prominent figures that unified Japan are: Oda Nobunaga, Hideyoshi Toyotomi and Tokugawa Ieyasu. Before the Edo period, during the warring states period, the wealthy elites/nobles/prominent families of Japan were not living the typical lives that one might expect; they wore simple clothing and didn't have m wealth; their situation was atypical for them.

When the warrior aristocrats came to rule Japan, they recognized the cultural capital that these elites held; they would have no power but were respected and held with high regards by the warrior aristocrats for their culture. The elites were told by the shogunate that their only duty was to contribute to the culture of Japan. And so they did; they did this through paintings, playing games, taking part in tea ceremonies as well as poetry.

Now, the shogunate was not too keen on the interaction between these elites and the warrior aristocracy; in fact, the elites were restricted to a walled off quarter. But, the interactions occured anyway (a whole other comment could be written on what the shogunate wanted people to do and what they actually did). The warrior aristocrats respected the culture that the elites had and some would even get tutored by them in how to behave like the elite class.

The samurai received rice stipends (think of it like their salary in a rice based economy) and were told to be moral exemplars for the rest of society and were expected to be cultured individuals.

So, despite having nothing to fight, the samurai were expected to uphold the ideas of Neo-Confucianism through this moral exemplary that would supposedly trickle down to everyone else; as well as refine themselves culturally. The samurai were a direct link to the Shogun himself so they would be higher than others simply due to that. But the order in this caste system was SPAM; samurai, peasants, artisans and then merchants (in order of importance).

The peasants were second most important for supporting the economy, the artisans for contributing goods were third and the merchants were seen as leeches who contributed nothing really.

To add on to what u/SerpentsGold said about Neo-Confucian influence, part of that ideological bundle which was so influential during the Edo period was the idea of the "four occupations:" gentry, peasants, artisans, and merchants. This four-class scheme first appears in Chinese Warring States-era sources, in a context in which the "gentry" (士 shi) were a class of specialists in government service, who held a variety of administrative, military and ceremonial roles. Over time, the shi gentry in China became demilitarized and identified strongly with civil government, but parts of the Classical Chinese corpus preserved aspects of the more warlike role of the ancient shi. Sima Qian's Records of the Grand Historian in particular has numerous biographies of exemplary warrior shi, and these tales of valor, loyalty and self-sacrifice were a more appealing model for the Edo warrior-aristocracy than the contemporary Ming system, which was both more genteel and more meritocratic. So while Edo Japan did look to Neo-Confucianism as a useful political and ideological model, they adapted it to Japan's particular social conditions by interpreting the ruling shi class in a more militaristic way than contemporary Chinese did.