The skull you're referring to is commonly called the Totenkopf and actually has a tradition long pre-dating the Nazis. Although it's not the exact literal translation, Totenkopf is often translated into English as Death's Head. While there is an obvious association with death, there seems to be no definite origin of this association.

The first uses of the skull in military insignia date back to the 18th Century. A regiment of Prussian Hussars was formed which used the insignia, and the British 18th Light Dragoons (later the 17th Lancers) also elected to use a skull as their cap badge at around the same time. The Royal Lancer Regiment still uses a stylised skull and crossbones with the motto 'or glory' underneath, the skull standing in for the word death. The skull and crossbones was also famously used as a flag by some pirate vessels.

The Totenkopf was in use by Prussian and after 1871 German military units up to and throughout the first world war. Particularly famous for this was Field Marshall August von Mackensen, who was honourary commander of Leib-Husaren Regiment Nr.1, one of the original regiments to bear the symbol. These Hussar regiments were commonly referred to as Totenkopfhusaren due to their distinctive insignia. Von Mackensen would continue to wear his old Hussar uniform throughout the Nazi period, making it very easy to notice him in photographs of official events he attended.

After the first world war ended, the Prussian Hussar regiments were amalgamated into the 5th Reiter Regiment, which continued the tradition of using the Totenkopf. The SS used the symbol from fairly early on in their existence, and once the Wehrmacht was expanded after Hitler's takeover of power, a number of units chose to use the Totenkopf insignia. Outside of the Wehrmacht, the 3rd SS Panzer Division also chose to use the Totenkopf. The SS-Totenkopfverbände (lit. SS Deaths Head Organisations), who were responsible for concentration and extermination camps became particularly associated with the symbol.

In summary, it wasn't every German soldier who wore the insignia, but select units. This was the continuation of a long tradition of use in the Prussian and German armies, not to mention widespread usage amongst individual units from other states.