The Talmud and Old Testament speak to the idea of a 'scapegoat'. The English translation of the Hebrew is "azazel". The scapegoat was the sender away of sins. See the book of Leviticus 16:8, "And Aaron shall cast lots upon the two goats: one lot for the Lord, and the other lot for Azazel".

Obviously, goats have horns.

In the Book of Enoch Azazel is a chief fallen angel.

By the time of Jesus, Azazel and Satan were associated together (see Apocalypse of Abraham and also the Book of John, John 12:31).

The idea of The Devil developed from this very early notion of the Scapegoat. Further, the Devil in most Christian theologies is a fallen angel.

In short, a whole group of ideas of evil beings got rolled up into one notion of Evil or the Devil. Therefore The Devil is a horned goat, a fallen angel, a tempter, a serpent, etc. Some of these ideas may also have been borrowed from Persian Zoroastrianism. Eventually the Roman Catholic Church had a highly developed and refined idea of who The Devil was and used this idea to scare the bejebus out of followers.

Apologies as this is not a very lengthy answer.