In 2010, Gizmodo wrote that the play symbol originated in the 1960s. It was by no means universal, 8 track tape players in the 60s often had no play button, simply playing when the cartridge was inserted. In the 70s, tape players increasingly featured buttons, sometimes with the words "play", "rewind", etc, other times with symbols, and sometimes a mix of both.

While the triangle's use as the play button goes back 50 years, it was not universal over that time. In the 1980s, Fisher-Price developed a line of audiovisual toys for preschool aged children. The tape player used a rightward facing arrow to mean rewind and play was represented with an eighth note. Moreover, in the Soviet Union, completely other symbols were found on audio equipment. Double arrows for rewind and fast forward were common, but even through the 1990s play was not represented with the familiar triangle.

Both the first Sony Walkman (the TPS-L2, released 1979) and Discman (D-50, released 1984) featured the familiar triangle and square for play and stop. Both of these products were seminal enough to become part of the collection of the Museum of Modern Art.

To piggyback on this question. It appears that the "record" symbol of a circle and the "eject" symbol of an underlined upward triangle and "pause" symbol of two vertical lines came later. Does anyone know when / those came about?