While people understood that the male royal monarchs of England had a long history of philandering, by this stage in history--arguably the height of social grace and decorum-- discretion was expected above all else. Royals may have kept mistresses but they were not above scandal when there was a lack of discretion. There are two factors that jump to my mind when considering Victoria's reaction: the fact that "actress" was synonymous with "prostitute" in Victorian England (which had to do with the continuous permeation of high society by increasingly middle class people, as well as the sometimes libertine nature of acting itself... actresses were prominent, often wore and said daring things on an off stage, and often started out in positions that put them in close quarters /dependent on men who were not their husbands/fiancees.)

Aside from that, royal "flings" with commoners could often turn more serious than just flings. Victoria's own contemporary, Elisabeth "Sissi" of Austria had a son and heir to Austria-Hungary who committed a murder-suicide with his own mistress Mary Vetsera. It was not the first time a crown prince took affairs too far, and Victoria likely had that in mind. Victoria's own relationship with Albert was a true match of equal rank, temperament, and love. It is generally known she dearly loved Albert. It's very possible that her grief at Albert's death gave her extra incentive to blame anyone and anything including her playboy son, whose affairs did not resemble her own ideal marriage.