Before the popularity of tabletop games like Dungeons and Dragons, what was the use of non-six-sided dice like D20s?

One of the uses of the D20 was for use in wargames and other tabletop simulations. Especially with the rise in wargaming with miniatures, skirmish wargaming, the D20 allowed for a more flexible use of probabilities than a standard D6. Using a D6 to gauge these probabilities often devolved into a math and recording drill. As Donald Featherstone's 1975 manual Skirmish Wargaming explained:

The 20-sided dice used in skirmish wargames are fundamental to the percentage system used in the rules as they permit all endeavors to be reduced to a clear percentage chance...The fluctuations of fortune are most adequately represented by the use of such dice- even top-level computerized wargaming by the services, dice are similarly used! If the rules indicate that a man has a 70 percent chance of hitting his target, when he throws the dice there is exactly a 70 percent chance of throwing a number between one and 70 which would indicate a hit. Conversely, there is a 30 percent chance of missing by throwing a number between 71 and 100. It is a simple and flexible system, offering many possibilities and presenting a calculated solution to any problem or situation.

As Featherstone indicated, the armed services had been using D20s for various wargames. The US Naval War College's Fundamentals of War Gaming by Francis J. McHugh had instructions and tables to incorporate D20s along with D6s as random variables and percentages. As with wargames on the civilian market, the D20 allowed for random percentages. Nor was this need limited to leisure markets. Icosahedral dice were a means to generate random numbers on a hundred scale in an era before cheap and portable computers. The Japanese Standard Association sold a number of these dice, including various colors, and was one of the more common venues to acquire these dice. Laboratories, research departments, and military organizations could acquire these dice and they gradually filtered into the civilian market by the late 1960s and early 1970s.

That the D20 entered into the gameplay of D&D was not surprising. The predecessor to D&D, Chainmail, was heavily influenced by the skirmish wargaming pioneered by people like Featherstone in the 1960s. But the growth of the RPG market was larger than wargaming and the demand for D20s outstripped traditional suppliers like the Japanese Standard Association. The result was the need for D20s and other specialized dice created a cottage industry for these dice with companies like Creative Publications selling them either as mail-order or to various hobby retailers after making investments in injection molding.