The words are describing the letter.

Epsilon means "simple e". It's the shorter one letter version of e sound, in distinction to the epsilon alpha-iota diphthong. Upsilon is the simple version of the omicron-upsilon diphthong.

Omega and omicron are just big and little versions of O. Everyone knows enough Greek to guess which is which.

It wasn't in Latin, but we have added "double U".

Most of these words were not current in Classical Greek, and were named as such in the Byzantine era, when many of the sounds had merged in common usage. So when ο "little O" "oo" and ω "big O" "ohh" started sounding the same, the Byzantine grammarians had to give them names that distinguished them in orthography.