For a very long time, at least as far as virility is concerned. I only have a somewhat superficial knowledge of Egyptian, but several words related to virility and fertility feature the testicle sign, although the relation to size I know nothing about

The earliest specific reference I can think of is in Aristophanes. In the Birds there are several jokes that refer to testicles, although it is not size that matters, but the number of testicles. The first is a pun on the word triarchoi, three rulers, and triorchoi, three-testicled, meant to poke fun at some events which have just occurred. The second, more dirty reference in the same play is when Pisthetaerus orders a buzzard to go seize Iris. The joke is pretty rude, implying sexual assault since the word for buzzard is triorchos, which we already saw means three-testicled and is a way in Greek of saying lecherous or excessively sexual active.

Now, when did virility get combined with valor? Difficult to say, but at a very early date. The linguistic terminology for virtue in many languages is often directly related to male fertility--for example, virtus in Latin. Additionally many cultures have a concept of the Begetter. In Latin for example this is the genius, from the verb gigno, to beget or create. The genius was the tutelary spirit of the head of the family and the rest of his clan and originally seems to have been in some way a fertility spirit, although it also governed the force of a man's character, including his bravery and loyalty.