I'm assuming you'll accept answers that look at more than just art history. It appears that this motif became most fully developed in Islamic tradition. For example, in the 50th sura (Qāf) of the Qur'an (vv. 16-18), the angels proclaim that
We are closer to [man] than his jugular vein––with two [angelic scribes] set to record, one on his right [side] and one on his left: he does not utter a single word without an ever-present watcher.
This motif was inherited from earlier ones - for instance, the early apocryphal Jewish text Testament of Abraham envisions the eschatological judgment scene of all individuals:
On the table lay a book whose thickness was six cubits, while its breadth was ten cubits. On its right and on its left stood two angels holding papyrus and ink and pen . . . The two angels on the right and on the left recorded. The one on the right recorded righteous deeds, while the one on the left (recorded) sins.
(It's also thought that much of this passage incorporates Egyptian judgment imagery)
We might also be able to discern a connection with other things. For example, in the New Testament Gospel of Matthew, it's said that during the eschatological coming of the Son of Man,
All the nations will be gathered before him; and he will separate them from one another, as the shepherd separates the sheep from the goats; and he will put the sheep on His right, and the goats on the left.
This may actual connect with other traditions involving angels.
It's proving extremely difficult to find the earliest occurrence of the angels actually on shoulders; but it could very well be that that this was simply an inference from the Qur'anic text (though nothing is actually said about shoulders here). Though I suspect this is quite modern. In any case, there's definitely a connection with traditional practice of salat - where prayer is directed to the right and left - ...though there's a bit of a chicken-and-egg problem here.
Follow-up: is this exclusively an idea with roots in the traditional Christian concept of angel and devil, or was the same concept ever used with different figures representing good and bad?
the literary term for it, i believe, is called psychomachia. it literally means "soul struggle" and comes from a poem 3rd century latin poet where vice and virtue battle. i cant speak to it's history in the art world.
source: i paid attention once in awhile in high school