In general, the surviving sculptures, vases, and writings describing male beauty in the Greco-Roman world depict the ideal of beauty for men (especially young men) as being hairless, thin, athletic, and with a small penis with a tapering foreskin. Large penises are most often found on satyrs, ugly old men, and barbarians, which suggests that they were viewed as grotesque, comical, and outlandish. In his study Greek Homosexuality, Kenneth Dover writes:

In caricature and in the representation of satyrs a penis of great size, even of preposterous size, is very common, and it is a reasonable conclusion (though not, I admit, an inescapable conclusion) that if a big penis goes with a hideous face, and a small penis with a handsome face, it is the small penis which was admired.

As for why this would be the case, we can only speculate, but Greek Homosexuality does a good job of arguing why and how Greco-Roman conceptions of sexuality were not the same as modern conceptions of sexual orientation. Sexuality was considered more fluid, especially in the case of sexual relationships between older men and adolescent boys, which were not seen to make either of the participants exclusively homosexual. Therefore a standard of male beauty evolved around what these older men desired in younger men, which tended to be a slim, hairless, somewhat feminine physique with a small penis, rather than a more virile or well-endowed standard based around, for example, female sexual pleasure or desire. It was the older men who largely ran Greek and Roman society, and who created and patronized most of the art and literature, so it is their concept of male beauty which we know about.

A PDF of Greek Homosexuality is here. I pulled that quote from page 126.

EDIT: I realize this answer is mostly about the Greeks when OP asked about the Romans, but from what I've heard the Romans had similar views on penis size as the Greeks, and Greek Homosexuality was the best academic source I could find about ancient Mediterranean dick size preferences.