The origins of the "Nine Dash Line," which effectively claim the entirety of the South China Sea for China, both the People's Republic of China (PRC) and the Republic of China (ROC), go back to 1947. On December 1, 1947, the Republic of China, which controlled most of mainland China at the time, published a map of China that detailed their claims over the South China Sea. At that time, it was an eleven dash line that included enlarged claims to areas in the Gulf of Tonkin. These were later removed after the Communist Party of China (CPC) fully took over the mainland to forge ties with the new communist government in Hanoi.

I'm not sure what you mean by "nebulous." In fact, both the governments in Taipei and Beijing maintain that the area is an inalienable part of China. However, it hasn't been until the last decade or so that the PRC has been bold enough to enforce those claims militarily, with several bases on the small natural and new man-made islands. The ROC also has its own naval base on Taiping Island. Both the ROC and PRC rejected the 2016 ruling by the Permanent Court of Arbitration at the Hague in the case of the Philippines v. China.

Both Taipei and Beijing maintain that the territory was historically part of previous Chinese-based states (notably the Qing) based on old maps. Quite frankly, the claims have little or no basis in reality. The Ming or the Qing may have, at various times, claimed that territory, but made no concerted efforts to enforce them. Both empires were almost wholly land-based, an effort to reduce both the power and influence of merchants and reduce foreign cultural political influences within their empires.

Moreover, the borders of these empires were nebulous at best, and wildly determined at other times. Mohan Malik, in his article "HISTORICAL FICTION: China's South China Sea Claims," notes that "In pre-modern Asia, empires were characterized by undefined, unprotected, and often changing frontiers. The notion of suzerainty prevailed. Unlike a nation-state, the frontiers of Chinese empires were neither carefully drawn nor policed bu were more like circles or zones, tapering off from the center of civilization to the undefined periphery of alien barbarians." In short, the South China Sea was never truly a fully integrated part of either the Qing or Ming, and these states made no effort to make it so.

The faulty logic justifying China's claims to the region is further complicated by the nature of the Qing and Ming empires in regards to polities along these nebulous borders. Both states maintained tributary relations with most of the polities along the northern and western borders of these empires, and across seas to the east. Most of these tributary relationships were established with nomadic or semi-nomadic Mongol, Jurchen, or Turkic tribes that lived on the steppes surrounding China. Of course, two key tributary relationships were established and maintained, off and on, with the kingdoms of Korea and the Ryukyus. The line between "tributary" and full-fledged imperial province was often difficult to determine.

The Ryukyus, in particular, have been pointed to by some Chinese historians as rightful Chinese territory for this exact reason. The logic is that the island chain, firmly officially Japan's, at one time was a part of a previous Chinese state's direct sovereign control (it was in fact ruled by a wholly separate state that sent tribute to the Ming and Qing) and was stolen by the Japanese Empire. While not directly related to your question, I think it helps illustrate the mismatch between historical and contemporary concepts of sovereignty, in short they are wholly different and largely incompatible. This is not something contemporary Chinese nationalists are wholly unaware of. One PRC general noted in 2016 that "Let's for now not discuss whether [the Ryukyus] belong to China, they were certainly China's tributary state. I am not saying all former tributary states belong to China, but we can say with certainty that the Ryukyus do not belong to Japan."

It bears strong similarities (at least at face value) to feudal European claims or land and titles, something that resulted in messy and often violent transitional wars in the 19th and 20th centuries. German nationalists, both in the 19th and 20th centuries, made claims using similar logic to Bohemia (now the territory of the Czech Republic), Alsace-Lorraine, and Austria on account these regions being historical parts of the Kingdom of Germany.

Chinese empires, particularly the Qing (although calling the Qing a "Chinese" empire is debatable) and Ming, similarly with feudal European states were not nations with a concept of Westphalian sovereignty. That logic of state borders simply did not exist in East Asia at the time and was broadly a Western import, although this was in part through several key wars between the Qing and European empires. In truth, even equating these former empires with the modern concept of "China" is problematic from a historical view, however that's another discussion altogether. The claims that China uses now to justify its desired takeover of the South China Sea assert that the logic of singular sovereignty of territory extended back to the Qing, which virtually all China historians disagree with. The islands, at the time, were uninhabited and neither the Qing nor Ming had any official naval presence there. The European traders that came later, similarly, saw little need to establish bases there, and it has only been the 20th century when states have attempted to exert direct control over the space. It could be argued that the enforcement of modern, international understandings of territorial sovereignty upon the Qing Empire during the 19th century robbed the Qing of its rightful claims in the South China Sea that were couched in wholly different logic, but largely this is not what Chinese nationalist historians are claiming.