So just to quickly correct some misconceptions before answering the question:
- Shōgun Ashikaga Yoshimasa was very much still alive at the start of the Ōnin War in 1467 and did not die until 1490, over a decade after the war ended in 1477, though he did retire mid-way through the war in 1473.
- As I explained here, while the Ashikaga succession crisis did heighten tensions between the two rival factions, it was not the immediate cause of war. The casus belli was the Hatakeyama clan's succession crisis. Only after the outbreak of war did it officially become about Ashikaga succession. The actual cause of war is the very complicated web of alliances formed around the Hosokawa and Yamana in their struggle for power.
- The Ashikaga Bakufu did not end with the outbreak of the Ōnin War. The Ashikaga clan continued to reside in Kyōto and be appointed shōgun, and the Bakufu continued to exist until Oda Nobunaga ran Ashikaga Yoshiaki out of town in 1573.
Now to answer the question.
The different "states" were already in existence. The Japanese word for the Sengoku states is the same word as the one for pre-Meiji Japanese provinces. That is 国 kuni. This might feel like arguing semantics, but what's clear is that even on paper, the 国 was the first level of administrative division of Japan, and so even in the chaos of the Sengoku, sides went by their kuni, and smaller factions had the unification of their kuni as their goal. For administrative boundaries, in a way the warring states were just extensions of peace-time provinces.
So the question is why did they suddenly spring up as independent forces? Well in a way they were not (yet). The Ōnin War was in effect over the control of the Ashikaga Bakufu, and each side ostensibly fought under the name of the Bakufu and followed the Bakufu's orders.
In another way, they already were independent forces long before the outbreak of the Ōnin War. As mentioned in the previous thread, wars and rebellions had already broken out before the Ōnin War, at least as far back as 1454, with the Bakufu unable to end the various fighting across Japan.
So why was the system so unstable?
- Changes in administrative structure through the Kamakura. The Kamakura Bakufu originally set up jitō and shugo positions in the various kuni to work alongside the central court's kokushi, who was ostensibly in charge of the kuni. The jitō was in charge of taxes and administration of realms besides the court's and manors, while the shugo the policing and military. This setup provided clear boundaries of responsibilities. However, the Kamakura Bakufu shielded their administrators from punishment by the court for encroaching on the powers of the kokushi. The jitō and shugo not only did so, but in an unintended consequence also increasingly acted independent of the Bakufu. By the late Kamakura, the kokushi existed only in name, with the jitō having taken over all taxes and administration, while the shugo took over various local judicial powers from the Bakufu.
- The foundation of the Ashikaga Bakufu was not very strong. While the Minamoto had wide control over Japan when the Genpei broke out, the Ashikaga did not. They were but one of the Kamakura Bakufu's vassals. This means the Ashikaga could not rely on its own troops alone to control Japan. Even when they were appointed (or more like declared themselves) shōgun, they depended on other families to provide the troops to help them fight through the wars of the early Muromachi period. Which lead to:
- The Ashikaga Bakufu's decision to implement the hanzei system. Literately "half-tax", the order was that the shugo was entitled to half of the tax revenue of his kuni. As the shugō had the troops, the Ashikaga Bakufu had to buy their loyalty. However, by giving the shugo an incredibly large financial base, the shugo, now having financial, military, and judicial control of a kuni, quickly became its de-facto ruler, and could in fact rule without the Bakufu. This lead to
- Everyone's concerns about the shugō. The shugo was now an incredibly powerful position, with the jitō having basically been rolled under his control. The Bakufu was very worried that shugo would rebel (and many did), and so mandated many to live in Kyotō. But this created another problem. With the shugo being absent most of the time, control of the kuni often fell to the deputy-shugo. And as the position of shugo was now so incredibly powerful, it attracted the ambitious to vie for the position, as family branches began fighting for the position of shugo and deputies rebelled sought to replace their shugo.
So yes, the Ashikaga Bakufu was unstable. But it was not just a single succession crisis (or two) that brought it down. It wasn’t that unstable. If I have time I’ll write something to summarize the convoluted mess that led to the Ōnin War. But hopefully this answers your question for now.