Why has Jesus not been looked at as merely a bastard?

There were accusations that Jesus was born from an adulterous affair. The most famous example of this is found in the Bible itself, in the book of Matthew:

Mary had been betrothed to Joseph, before they came together she was found to be with child of the Holy Spirit; and her husband Joseph, being a just man and unwilling to put her to shame, resolved to divorce her quietly.

Obviously and understandably, Joseph (prior to being dissuaded of the notion by an angel) believed that Mary had committed adultery.

These rumors appear to have persisted, and in the 2nd century Celsus, the greatest pagan critic of Christianity, accused Jesus of being the result of an adulterous affair of Mary and a Roman soldier. From Contra Celsum I.32:

when she was pregnant she was turned out of doors by the carpenter to whom she had been betrothed, as having been guilty of adultery, and that she bore a child to a certain soldier named Panthera

A similar account also appears in the Talmud. Although some argue that the references in question do not actually refer to the Jesus, but some other man of the same name (I am not qualified to judge). It might be notable that Celsus places his accusations about Jesus's heritage in the mouth of a Jew, suggesting that he might be repeating rumors widely circulated among Jewish communities at the time. So yeah, people did accuse Jesus of being a bastard.

So there are two questions here:

1: Why haven't more people assumed Jesus was a bastard?
2: Is the story of Jesus' virgin birth evidence of Jesus being born out of wedlock?

-- /u/wedgeomatic answered the first question, but I don't think anyone's answered the second one yet.

The short answer to the second question is no, modern scholarship does not consider the story of the virgin birth to be evidence Jesus was a bastard. The long answer requires some knowledge of Biblical criticism.

When analyzing any ancient text, we have to use tools of critical analysis to determine what likely happened and what didn't. Many ancient sources are mixtures of myth and real history. For example, Herodotus (~450BCE) talks about men without heads who have eyes in their chest in one passage and then fairly accurately describes Scythian burial rituals in another. Some of the critical tools we can apply to determine the veracity of a passage are:

1: Are there multiple independent sources that agree with one another?
2: Is the source close in time and place to what it is describing?
3: Does the passage agree with the ideology of the writer? (for example, if a Microsoft spokesperson said "the Xbox is great" that would tell us nothing about the Xbox, but if a Sony spokesperson said the same, it would probably have more truth value)

In the case of the New Testament we have a few sources: the letters of Paul and the Gospels of Matthew, Mark, Luke, and John. Paul is the oldest source and doesn't mention the virgin birth. Mark is the second oldest source and doesn't mention it either. Luke and Matthew both mention the birth, but they disagree with each other about the details:

Luke 1:26-30: In the sixth month the angel Gabriel was sent by God to a town in Galilee called Nazareth, to a virgin engaged to a man whose name was Joseph, of the house of David. The virgin’s name was Mary. And he came to her and said, "Greetings, favored one! The Lord is with you." But she was much perplexed by his words and pondered what sort of greeting this might be. The angel said to her, "Do not be afraid, Mary, for you have found favor with God. And now, you will conceive in your womb and bear a son, and you will name him Jesus."

Matt. 1:18-20: Now the birth of Jesus the Messiah took place in this way. When his mother Mary had been engaged to Joseph, but before they lived together, she was found to be with child from the Holy Spirit. Her husband Joseph, being a righteous man and unwilling to expose her to public disgrace, planned to dismiss her quietly. But just when he had resolved to do this, an angel of the Lord appeared to him in a dream and said, “Joseph, son of David, do not be afraid to take Mary as your wife, for the child conceived in her is from the Holy Spirit."

In Matthew, an angel comes to Joseph in a dream, in Luke, the angel Gabriel physically comes to Mary. If you continue reading, you'll find that Matthew and Luke continue to differ in almost everything until they get to Jesus' baptism. So let's use the critical tools above: The earliest sources do not mention the virgin birth (point 2), the sources that do mention it disagree (point 1), and the virgin birth agrees with the ideology of the writers (it "took place to fulfill what had been spoken by the Lord through the prophet" (Matt. 1:22)) (point 3).

So, there is very very likely no historical basis for the virgin birth. Jesus was probably one of several children of Joseph and Mary. It was most likely a later tradition originating from Jewish ideas at the time, but possibly influenced by the other religions in the Greco-Roman world.