Yes, it is possible. What is more interesting is that no one seems to have cared about the colour of his skin for about 1300 years thereafter, possibly even longer. In fact we will never know, and our obsession with Victor's skin colour tells us more about ourselves than about him.

Have a read through of this, which although is about the middle ages is about the same for Romans.

Pope Victor was born in Roman Africa, which encompasses Northern Africa and Egypt. This region in this time period, as it is today, was predominately white or fair skinned. The chance that he might have been darker or tanner of skin is possible, but he would've looked more like an Arab or Berber than, say, a South African.

Just curious; what prompted you to ask this question?