So, I can speak to this a little from a historical perspective of the ancient world. Perhaps someone else can chime in with medieval perspectives.

First off, let's disentangle some terms in their original context.

"Satan" in Hebrew means "adversary" or "accuser." The sense is sometimes used formally as "prosecuting attorney." The term "Satan" shows up in the Hebrew Bible only a handful of times. The only clear reference to a character is in the book of Job (and maybe Zechariah). There are vague and debatable references in a few places that may just mean "an adversary." I've heard Bible scholars say "HaSatan" (stress on last syllable) to distinguish the entity we're discussing, because the definite article ("Ha") is only present in the two clearer cases of Job and Zechariah.

In Job, Satan is a member of God's heavenly court, a spirit or god or angel of some kind assigned to accuse humans. The relevant section is Job chapters 1 and 2. Satan seems to want to catch Job in the act of sin, while God wagers that Job will stay faithful, no matter what harm Satan may do to him. God and Satan aren't depicted as the best of friends, but there's a clear sense that Satan works for God, answers to God, and has to get permission from God to afflict Job.

Lucifer comes from Isaiah 14, and refers to the King of Babylon. It doesn't mean Satan.

There is no "Hell" described in the Hebrew Bible. The word "hell" is a Germanic borrowing, but the New Testament mostly talks about Hades or Gehenna. Gehenna is a real place, a location of child sacrifice. The Hebrew Bible refers to it several times in this capacity, deploring the ritual of infant sacrifice. The New Testament seems to use it more or less interchangeably with Hades. There is contradictory evidence as to how much this was imagined as a place of suffering, or simply a state of anihilation. On the one hand, there are a few places that refer to the "gnashing of teeth" etc, but just as commonly the early Jesus movement talked about their goal being "Life" or "Everlasting Life," as opposed to death and hades.

Satan shows up with a few names and in a few roles in the New Testament, somewhat closer to the contemporary idea of a "devil." The term "diabolos" is used for him, which is a Greek word for an accuser or slanderer. In the Tempation narrative, Satan mostly acts (as in Job) as an accuser, trying to trip Jesus up. But elsewhere we get some sense of him as a leader of demons. In a few places the roles seem to blend into each other.

The probable causes of the shift in thinking about Satan in relation to demons (the origins of which merit another long post) or as an incarnation of evil, or cosmic adversary of God are Zoroastrian and Gnostic dualism. Angra Mainyu in Zoroastrianism is very similar to many traditional ideas about Satan. Angra Mainyu is a spirit of chaos and destruction. You can't really be a "follower" of Angra Mainyu, you can just be chaotic, destructive, and cruel in your own life, which leads to both your own destruction and harm to others. Angra Mainyu, insofar as we can assign him/her/it agency, is interested in the destruction of the created order and all that Ahura Mazda has made. The wicked are not friends of Angra Mainyu; they are those damaged most in the effort to undo creation.

This question isn't inappropriate for this subreddit, as there is certainly an historical answer to be had regarding where this belief came from, but you may also want to ask it at /r/AcademicBiblical, where the theological implications you bring up can be explored.