Romans used them as letters also. Greeks used their letters as numerals as well. It's not especially odd. One Latin word for army is exercitus. Would have been written as EXERCITVS. There you get all three of those symbols used as letters in a common word.

It does take some practice to learn context. Sometimes someone holding the office of triumvir will abbreviate on coins or inscriptions as IIIVIR - that's not pronounced iiiiiiiiiivir or meant to be 36r; it's trium (iii - 3)vir (man) or a member of a board of 3.

It's not hard to tell when those signs are numbers or letters when you get used to them.