More can always be said, but in the mean time, /u/commiespaceinvader has written some previous answers on the topic of Yugoslavian identity and the break-up of the country which you might find interesting.
This one focuses on the build-up to that point, looking at Tito and his leadership, while this one focuses on the underlying idea of "Balkanization", and this follow-up in particular being perhaps most relevant for the direct question, as it looks to the '80s after Tito's death.
/u/commiespaceinvader has great writeups on the subject, as always, but I think I can provide some additional information as well.
To a much larger degree than frequently acknowledged, the conflicts between the nations of the Western Balkans are religious and deeply historical in nature. In Western media during the Balkan Wars, the breakup of Yugoslavia was frequently presented as a logical, if regrettable, division along ethnic lines. To a certain extent, this must be true, as the people of Serbia and Croatia came to view themselves as separate nationalities. This view of the situation is very facile in nature, however, and a deeper look at the situation shows how the people of the Western Balkans came to develop a pan-south Slavic identity, and how historical and religious factors ultimately helped to drive a fatal wedge in the Yugoslav experiment.
In my answer I'll primarily focus on the differences between Croats and Serbs, both because I think those differences are the most relevant to a discussion on the breakup of Yugoslavia and because that's the area where I'm most knowledgeable.
Ethnicity always has an element of social construction, but the ethnic differences between the different peoples of the Western Balkans are even more socially constructed than most. Like every assertion made regarding Serbs and Croats, this will provoke strong reactions, as every such assertion involves political and national considerations. You can easily find Croats and Serbs who will say that their people are only as similar as Croats are to Poles. An attempt to find unbiased information on the internet about the legitimate differences between the groups will be extremely fraught, with a lot of propaganda.
Genetically, there is very little difference between a Croat, Serb, and Bosniak. Genetic studies have shown close affinity between the three groups. Popularly, there may be a conception that Bosniaks, due to their Muslim faith, have substantial influence from Turkey but this is largely untrue. Nearly all Bosniaks were converts to Islam from the existing South Slavic population during the Ottoman period. Frequently this occurred due to the advantages afforded to Muslims by the Millet system. The three peoples are far more similar than other groups considered a single ethnicity.
In terms of language, Croatian, Serbian, and Bosnian are mutually intelligible and can be regarded for most purposes as a single language that has seen increased variation and a decline in intelligibility since the breakup of Yugoslavia. This is still controversial in linguistic circles, because a substantial portion of Croatian linguists will claim that Croatian and Serbian have always been different languages, with periods of greater overlap (such as during the existence of Yugoslavia). On the other hand, Serbian linguists more frequently propagate the view that the language of the South Slavs is best described as Serbo-Croatian. Within this discussion, one can see the acrimonious legacy of nationalism and the breakup of Yugoslavia. Without taking sides on a complex linguistic debate, what I will say is that the linguistic variation of Yugoslavia was not as extreme as many countries which have strong national identities. A good point of comparison might be between Portuguese spoken in Portugal versus Portuguese spoken in Brazil. The uninitiated may have some difficulty understanding and being understood, but someone from Croatia should have no problem communicating with someone from Serbia after a period of adjustment. They are certainly more closely related than Spanish and Catalan, for example.
I felt as though I needed to clear away some of the major misconceptions about the different South Slavic groups which composed the people of Yugoslavia before getting to the meat of the answer. The salient distinction between the people of Yugoslavia has always been religious in nature.
Without getting into a deeper discussion of the factors which led to the adoption of Catholicism by the Croats and Eastern Orthodoxy by the Serbs, what can be said is that the Western Balkans were long the area where the influence of Latin West and Greek East clashed- a divide that can be seen in the usage of Cyrillic for writing in Serbian and Latin for writing in Croatian. This divide manifested itself in the religions adopted by Slavic groups in the Western and Eastern portions of the Western Balkans.
Both Serbs and Croats were long administered by plurinational Empires, Austria-Hungary in the case of Croats and the Ottoman Empire with Serbs. Though both were plurinational, each had different manners of handling the various groups of people over which they ruled. Austria-Hungary saw more homogenization and assimilation, to such a degree that in many parts of Slovenia (prior to the revivals of the 19th century romanticism and nationalism) German replaced Slovenian as a daily language. In contrast, the Ottoman Empire administered using what’s termed the millet system. Under the millet system, the laws and customs of an administered group were determined by the religion of their people. Throughout the history of the Empire the amount of autonomy fluctuated, and the conception of a millet varied, but the millet system allowed for a distinct Rūm millet identity to form among the Orthodox subjects of the empire. Crucially, a distinct Serbian identity coalesced which was separate from the groups wider identification with the Rūm millet.
A central Serbian state had never quite existed prior to the Ottoman conquest, but there were various independent Serbian principalities which had come together under a larger Serbian banner to fight against the Ottoman threat during the late 14th century. Serbian forces, led by Prince Lazar, faced the Ottomans in the Battle of Kosovo in 1389. Reliable historical accounts of the battle are scarce, and the mythos surrounding the battle overstates its decisiveness in the war against the Ottomans, but what is important regarding the Battle of Kosovo is how it influenced the development of a Serbian identity. The Battle of Kosovo looms over Serbian national consciousness to the present. It is frequently interpreted as a valiant last stand against the Muslim invaders in an attempt to defend the Christian world against an invading “horde”. The religious aspect of the Battle of Kosovo cannot be understated, as it came to be celebrated as “Vidovan” or Saint Vitus’s day, a Christian and national holiday where the Kosovo myth was propogated. One can write an entire book about the Kosovo Myth itself, but at its core it is a reinterpretation of battles lost by small principalities into the overarching national mythos of Serbia itself. Vidovan celebrates the sacrifice made by the heroes of the battle as a martyrdom in defense of Christiandom, a sacrifice of earthly freedom and power which led to eternal glory for the martyrs in heaven. It castigates the betrayal of Serbs who did not similarly sacrifice themselves in defense of the homeland. It is hard to understate the extent to which this conception of the Battle of Kosovo has influenced world events. In 1876 when Serbia declared war against the Ottoman Empire, they did so on Vidovan. In 1914 when Gavrilo Princip assassinated Franz Ferdinand, he did so on Vidovan. And most pertinently to the original question, in 1989 on the 600th anniversary of the battle, Slobodan Milosevic delivered his Gazimestan speech on the site of the battle. Religion and national identity are so tightly woven together in the Kosovo myth as to be entirely inseparable. The separate religious millet for Serbians and the legacy of the Kosovo myth led to a budding Serbian national identity which existed for much longer than a corresponding Croatian national identity. As in the rest of the world, this budding identity exploded in the 19th century, but it had been festering for hundreds of years. Serbs saw themselves as the protectors of Western Europe who had sacrificed themselves to save Christendom. As a result they came to view themselves as the rightful leaders of a larger Serbian nation which includes Croats, Bosniaks, Slovenes, and Montenegrins, an idea which they have never truly relinquished.
Continued below ------>
I disagree with the premise of your question on two counts.
Let's first look at this part: "but its creation was extremely popular post-WWI".
This is a perspective strongly slanted in favor of Serb and to a lesser extent Montenegrin voices and mostly ignores the other constituent ethnicities. I will mainly take the perspective of the Croats here, since the Croats were the #2 ethnic group behind the Serbs throughout Yugoslavia's history. But I am well aware that I am, for the sake of simplicities, going to have to skip over the plight and oppression of the other ethnic groups suffered at the hands of both the Serbs and Croats during that time, including the Slovenes, Macedonians, Bosniaks, and especially the Kosovo Albanians. I can follow up on the perspective of any of these groups if interest persists.
#The concerns of non-Serb ethnicities, particularly the Croats, in the leadup to the formation of the Kingdom of Yugoslavia
I'm careful not too bash the Serbs too much (there is something to be said about western voices being almost naturally biased against the Serb perspective in favor of their Croatian adversaries) whenever questions about the KoSCS/Kingdom of Yugoslavia come up (1918-1941), but the society was an extension of the Kingdom of Serbia of 1914, and was perceived as such by the other constituent peoples. This holds true for the ethnic Germans, ethnic Hungarians and ethnic Italians that lived as minorities in different parts of the kingdom, but also for the so-called "Habsburg Slavs", which is a grouping of the Slovenes, Croats and to a lesser extent the Bosniaks (who are usually excluded because Bosniaks are majority Muslim, whereas the term "Habsburg Slavs" is more strongly associated with the Roman Catholicism that connected the Slovenes and Croats to their former Austrian overlords).
The mindset of the Slovenes and Croats is the result of a unique dilemma:
On one hand, they were denied direct political representation at the levels of ethnic Germans and Hungarians in the Austro-Hungarian dual monarchy and were instead shoved into Cisleithania, the German-speaking part of the realm, forced to be content with minority voices in the German-dominated assembly. Even among the Slavs, they were subordinate to the West Slavic Czechs, who also sat in the same parliament and enjoyed greater political and economic importance. This puts the Slovenes and Croats in contrast to the Serbs, who established their own kingdom's independence from the Ottomans during the Ottoman era of weakness that started in the early 19th Century and that was marked by the independence of Greece, Serbia and later Bulgaria.
On the other hand though, the Croats and Slovenes saw economic upsides from their membership in the Habsburg realm. Their lands were more industrialized and more economically productive, and the people were supported by the Habsburg government in the practice of their chosen Catholic faith, whereas a Yugoslav project in coalition with Belgrade would inevitably mean the advance of Orthodox clerical interests.
And let's not be too quick to just lump all of these peoples together as "Yugoslavs" and call it a day. This is a result of a sometimes regrettable tendency to group ethnicities together with each other in ethnic families and to view these ethnic families as coherent units predestined to harmonize. That is not to say that these ethnic families didn't play a role, because they did, and the family of South Slavs especially inspired some early thinkers among both the Serbs and the Habsburg Slavs. But these thinkers were overall in the minority.
The conflicts between Croats and Serbs, particularly religious disagreements, made sure that there was almost unhealthy amounts of distrust between the two ethnicities, and that distrust has since flared into open rivalry and hatred over the mistreatment and oppression of Croats by the Serbs in the 1930s, over the alliance of the Croatian fascists with the Nazi invaders (and subsequent genocide of ethnic Serbs) during the 1940s, over the opposing political mindsets of the two peoples on the identity of the Kosovo Albanians, and of course over the Yugoslav Wars of the 1990s. But these grievances only came about during or after the creation of the Yugoslavia project of course, so let's scope back to the late 19th Century and the beginning of the 20th Century.
The Croats, and to a lesser extent the Slovenes, felt adversarial to the Serbs even during the times of Habsburg rule, partially because of Serb (which at that time was mostly defined by the trait "religiously Eastern Orthodox") minorities living among them.
Alex Dragnich notes:
Although Croats and Slovenes, unlike the Serbs, did not have their own national parliaments, they did have political groups or parties in which Serbs from those areas participated. In the ranks of these parties came some early divisions between Croats and Serbs, which contributed to future difficulties. In Serbia political parties differed in terms of ideas and programs. They did not divide along nationality lines. In Croatia, however, they divided along Serbian or Croatian orientations, toward Belgrade or toward Zagreb. Although the Serbs and Croats of Austria-Hungary did create the Croat-Serb coalition, it was significant that it was not called a Yugoslav or South Slav coalition. To the Croats it seemed crucial to retain their identity, and the drive to preserve that identity played havoc with many a pro-Yugoslav orientation in Croatia. On the other hand, when the Serbs considered a Yugoslav state, they thought in terms of melting into a new nationality - Yugoslav.
As history will show, the attitude of the Serbs towards this united Yugoslav identity would later change, but the mindset did at that time persist. The Serbs, who lived in an ethnically homogenous country, saw little issue with expanding that identity to other Slavs, who they felt understood the Serb perspective on oppression by a foreign power. But from a Croat autonomist perspective, the Serbs were just yet another oppressor waiting in the wings, aiming to replace the dominion that the Germans held over Croatia with a Serb dominion over Croatia.
#World War I
But these cultural concerns were forgotten in the heat of the moments of 1908 and 1912, when first the Bosnian Slavs were annexed into Austria-Hungary, and then when the Serbs and their neighbors united against the Ottoman Turks to fight the First Balkan War. For a brief moment, Yugoslav sentiment flared very high in both the free Balkan states and in Austria-Hungary, especially among Serbs living under Austro-Hungarian rule. One of those Serbs was of course Gavrilo Princip, whose assassination of the Austrian heir apparent Franz Ferdinand on 28 June 1914 triggered the immediate leadup to World War I.
Although Serbia was defeated in that war, its Entente allies won through to victory, and it was clear that Serbia would be reestablished in the postwar order. But we didn't end up with just a Serbia, we ended up with a "Kingdom of Serbs, Croats and Slovenes" (Kraljevina Srba, Hrvata i Slovenaca - hence often abbreviated SHS Kingdom), headed by the former King of Serbia Peter I (r. 1903-1921) and then his son Alexander I (r. 1921-1934).
So how did this happen after I just told you that the Slovenes and Croats weren't too hot on the idea? Well, because the Serbs forced their hand. And again, I am mindful to not just devolve into Serb-bashing here. It should be pointed out that the Serbs suffered the greatest losses in World War I of any country as percentage of the population, and that they did it in what they believed to have been the defense of all Yugoslavs, not just of the Serbs.
So, Marie-Janine Calic writes about the decision on Yugoslavism in 1917 as follows:
As for the structure of a future Yugoslav state, the question remained undecided. In 1917, three groups, each with a different agenda, stepped up to be heard.
The first was the Serbian government in exile on Corfu, led by Nikola Pašić, which favored a Serb-dominated centralist state that Slovenes and Croats would merely join.
The second was the Yugoslav Committee in London, which demanded a federal constitution based on the right to self-determination and thus equality among all three participating peoples.
The third was the Yugoslav Club, an organization formed by Slovene, Croat, and Serb members of the Austrian parliament (Reichsrat) for the purpose of finding a solution within the framework of the Habsburg state. On 30 May 1917, the club called for the "unification of all countries of the monarchy" in which "Slovenes, Croats, and Serbs live ... on the basis of the national principle and the Croatian constitution." The May declaration electrified the South Slavs of the monarchy, sparked public rallies, and inspired signature petitions for the national program. However, the declaration soon outlived its political relevance, and even the club abandoned its loyalty to the Habsburg emperor early in 1918.
[Reformatted, but otherwise unchanged]
So, you have the model of Serb-dominated central monarchy, the model of egalitarian confederalism between the SHS ethnicities (again, the perspective of the Bosniaks, Macedonians and Albanians gets ignored, and the Montenegrins were just shoved in with the Serbs), and even a model in which the Habsburg Slavs would remain with Austria under a triune model, forming an Austria-Hungary-Slavonia or something. And this latter model wasn't as unpopular as you might think. As I said, the Habsburg symbolized stability, economic prosperity, and the enforcement of the Catholic faith, and they were juxtaposed with the belligerent, unstable, poor and religiously Orthodox Serbia as an alternative.
Part of my studies involved looking at this problem in comparison with the collapse of the Soviet Union, so I'll provide what I uncovered in my research. What's important to remember is that there are generally two key theories that historians of this region debate on, which is whether Tito was the "incubator" of nationalism by policies that saved unity by doing a "give and take" between one state in the federation and another; or if Tito was the "freezer", as in he "froze" Balkan nationalism, which "thawed" after his death due to a lack of a clear successor.
To begin, while the dream of a pan-Slavic state was popular in the Balkans both prior to and during WWI, when the "Yugoslav Committee" was formed in London during the war, disagreements quickly emerged between the delegation from Serbia (which wanted a centralized state) and delegations from other nationalities (which wanted a more federal structure with devolved powers). When the Kingdom of Serbs, Croats, and Slovenes was established after WWI, it never really solved this divide in its initial governmental structure. Belgrade held some powers, but some power was also given to constituent regions. Additionally, ethnic tensions flared up soon after the war, with nationalists like Stepan Radić of the Croatian People’s Peasant Party calling for an end to Serbian dominance and a continued problem with the Serbian-controlled security forces routinely stepping into elections. Six cabinets came and went from 1918-1921, and nothing of substance could really be passed into law. The assassination of Radić by a Serbian nationalist in 1928 certainly didn't help, nor did the passage of a new constitution by King Alexander I in October 1929 which transferred several regional powers to Belgrade, strengthened police powers, increased press censorship, and banned any parties from having an "ethnic or religious slant". In the 1931 elections, the only parties running were the pro-Serb parties that had been approved by the government.
During WWII, while Yugoslavia initially made moves to side with the Axis, King Peter II's rise to power stopped this and led to the Axis invasion of Yugoslavia in April 1941. Germany installed the Ustaša Party (clerical Croat fascists) into power in a new collaborationist state called the "Independent State of Croatia" (NDH). Under the NDH, the Holocaust was willingly and eagerly carried out, with its targeted victims also including Serbs and Bosnian Muslims. Estimates on Serb deaths are around 200,000-500,000. Tito and his Partisans liberated Yugoslavia in 1945, and began their rule in 1946.
Under Tito, Yugoslavia was quite decentralized in comparison to its communist neighbours. Local and regional leaders made many decisions independent of Belgrade, and republics had "veto power" over certain legislation. Tito mostly controlled defence and foreign relations. Under Tito, party purges still happened. Liberal Serbs and Croats in the ruling League of Yugoslav Communists were purged in 1971 for calling for more decentralization. Regardless, Tito always enjoyed consistent levels of popular support and was seen as a "unifier". However, he never selected a successor or trained anyone to fill his shoes, so his death in May 1980 led to a sort of haphazard rule with presidents being routinely up for re-election (due to very short term lengths) and few pieces of legislation actually passing (similar to 1918-1921). The younger generation of communists also focused more on economic problems and relied less on the police/security services to limit nationalist sentiments in the press/public discourse. Leaders like Milosevic also broke party tradition by speaking openly on ethnic issues, as seen during his 1987 trip to Kosovo where he vocally supported Serbian issues there and began calling for a renewed centralized structure based on rule by Belgrade. The 1990 LCY convention was mired in ethnic problems, and soon after its conclusion, republics voted on independence and civil war began.
So to answer your question, the death of Tito and a lack of a clear successor was the catalyst for which regional nationalism really began to surge. However, "incubator" adherents often point to Tito's rule in and of itself as being flawed because, as Norman Naimark best put it:
...Tito's penchant for maintaining balance between the nationalities by giving to one and taking from the other, depending on the time and circumstances, ended up antagonizing all of them eventually and exacerbating resentments among them (Fires of Hatred, 146).
Additionally, many of the problems that plagued Yugoslavia at the end of its existence were fundamental disagreements that had precipitated at its start, and certainly hadn't been helped by ethnic violence both during the interbellum and WWII.
Sources: Benson, Leslie. Yugoslavia: A Concise History. New York: Palgrave Macmillan, 2004.
Ramet, Sabrina P. Balkan Babel: The Disintegration of Yugoslavia from the Death of Tito to the Fall of Milosevic, 4th edition. Oxford: Westview Press, 2002.
Vladisavljević, Nebojša. Serbia’s Antibureaucratic Revolution: Milošević, the Fall of Communism and Nationalist Mobilization. New York: Palgrave Macmillan, 2008.
Naimark, Norman M. Fires of Hatred: Ethnic Cleansing in Twentieth Century Europe. Cambridge: Harvard University Press, 2001.