Followup question: Since Joseph did marry her before Jesus was born, what would the popular perception have been in Israel when she bore a child who'd obviously been conceived prior to the wedding?
(Assuming/pretending they did return to Nazareth after Jesus' birth, that is - several popular Christian writers speculate they stayed at Bethlehem to avoid gossip, which would explain why Matthew places them there two years later when the Magi arrived.)