Originally, the major symbol of Christianity was the Chi-Rho. At the time, many people were crucified, and the crucifixion of Christ did not seem to be the most important detail of his life. As said by Everett Ferguson, a professor of church history emeritus at Abilene Christian University,
Christians did not make explicit pictures of the crucifixion for about 400 years after Christ's death. There are various possible explanations for this. A theological reason may be that the early Christians emphasized the resurrection as well as the crucifixion. (The Romans crucified many people, but only one was resurrected. Allusions to the resurrection in early Christian art were also relatively rare but earlier than depictions of the crucifixion.) A more practical reason may be that Constantine's influence so established the chi-rho that no alternative symbol was necessary. It functioned very much like a bare cross does today as an identifying marker of Christianity.
In the early days of Christianity, images included peacocks, doves, and particularly fish. The greek word for fish, ichthys, served as an ancronym for Iesus Christos Theos Huios Soter ("Jesus Christ, Son of God, Saviour").
In the second century, Christians used biblical images such as Noah's ark, Abraham killing the ram, and the raising of Lazarus.
When they dis choose imagery, as said by John Stott, Christians wanted to:
commemorate as central to their understanding of Jesus neither his birth nor his youth, neither his teaching nor his service, neither his resurrection nor his reign, nor his gift of the Spirit, but his death, his crucifixion.
At this time, the cross was not a crucifix, but rather a bare cross without christ hanging on it. These images arrived in the sixth century.
Stott notes that he believes the cross was in use by the end of the second century. He interprets Tertullian's AD 200 writing, "we trace upon the forehead the sign," as being the sign of the cross (De Corona, chapter 3). He also presents the writing of Hippolytus (The Apostolic Tradition, AD 215):
He mentions that the sign of the cross was used by the bishop when anointing the candidate's forehead at Confirmation [...] [i]t is also, he adds, a protection against evil.
COnstantine encouraged the use of the cross after he says to have seen a vision in the Battle of the Milvian Bridge. The vision showed a cross and the words, "Conquer by this sign." The victory inspired Constantine's act of conversion, and he began using the cross on the standards of his army. The cross thus became widely known across the Roman Empire as the symbol of Christianity.
Sources:
John Stott, The Cross of Christ
http://www.christianitytoday.com/ch/asktheexpert/ask_whythecross.html
http://www.historytoday.com/richard-cavendish/battle-milvian-bridge