There's a real paucity of sources to attitudes towards masturbation during the Roman pre-Christian era; what we can gather is that male masturbation was probably thought as morally neutral other than being somewhat dirty (i.e. it was something you did with your left hand, which was reserved for to be used in toilet and for other unclean everyday things), so probably not much different from today. As far I'm aware there's really not any sources that mention female masturbation; I'd imagine the attitude towards it was somewhat stricter at least for higher classes, since ideally women should not have a sex drive, though I don't imagine most Romans would have seen it as horrendously bad vice as long as it was not done in excess nor in the open. The idea of masturbation as something reprehensible and morally bad didn't come about in the West until Late Antiquity, when St. Augustine argued that sex and sexual pleasure was sinful in any other context than for conceiving children, which quite drastically shaped the later sexual mores of the Christian West.

In regards to Vestal Virgins, none of our sources discuss whether female masturbation was required from their celibacy, so, it's likely that it was not or no-one ever thought to ban it. The official Roman theory for why the Vestals had to be virgins is because they were considered the 'daughters of the State' so having sex with any citizen would have been incest. So, maybe there was no theological problem for Vestals helping themselves out. Of course, their sexual purity was also symbolically linked to the well-being of the state, and there's a couple of instances in Roman historiography where things were going bad for the Romans, and then it emerged that the reason for this was that one of the Vestals had broken their vow of chastity. But, no Vestal was ever punished for masturbation, that's for sure. Punishing Vestals was almost never required: we must have only a dozen or so cases where Vestals were but to death during the 1,000-year-history of the institution.