Much like the other groups you listed, the origins began through prejudice and were reinforced through various forms of fiction.

Before going into the gendered perspective of the "evil magic" trope, I want to note that, as with most racial stereotypes, the history of anti-Romani racism is a familiar entanglement of fear and hatred toward the "Other" as well as fetishizing the exotic. For a more detailed dive into this aspect, Ken Lee's article on "Gypsylorism" - a variation of Edward Said's seminal work on "Orientalism" - is a good place to start.

One of the most visible roots of the "evil magic" myth is at the intersection of Romani women, traditional practices, and the fortune-telling trade. Fortune-telling in Romani cultures is practiced both within the community and outside of it; the exact practices and spiritual traditions can differ from group to group and from family to family (i.e. Tarot and crystal divination don't have Romani origins but I've seen families who have adopted them into their closed practices). However, many of the practices in herbalism, palm reading, and card reading have entwined Romani women with the same "witch" archetype of other women of color - think along the lines of indigenous "curses" and black women practicing Voodun. Cultural and spiritual traditions are reinterpreted, especially through a Protestant Christian lense, as evil or demonic. (In actuality a majority of religious Roma are Orthodox or Catholic; culture and spirituality are interwoven with Christianity for many groups, for example with Sara e Kali, the patron saint of Roma.) As an aside none of these traditions are pagan or have anything to do with pre-Christian European religions. If anything most of them seem to have their roots in the places and empires along the route proto-Roma traveled on their way to Europe.

Fortune-telling is a traditional trade for Romani women that is a generally acceptable way to earn a living and interact with the non-Romani world. I tend to always refer to Alexandra Oprea's sentiment that, in a sense, fortune-telling can serve as a form of therapy where one person is seeking spiritual guidance or is simply looking for an impartial party to talk through their thoughts and emotions. It serves practical purpose as well as spiritual: Romani women could not only earn extra money for their household, it was a trade that was portable (which was essential as most Roma men worked in migrant trades such as metal smith in different towns, seasonal farm work, and horse trade to name a few), and they could schedule it around other household or family duties.

Historic evidence points to many of the fears surrounding the "evil magic" myth as a stereotype that evolves with time and area; in the 1450s, it seemed to be concerns about Roma being spies for the Turks and progressing into Christians fears of the occult through fortune-telling (more elaboration can be found in Role of the Romanies: Images and Counter-Images of 'Gypsies' in European Cultures). In my own research focusing more on the 18th-20th centuries the "evil magic" is tied to notions of criminality as well. In some cases the "magic" was the crime, most notably in examples of Romani women arrested and deported to the American colonies and Australia for fortune-telling. In others, fortune-telling and magic were the means to an end in either fraud or kidnapping. One newspaper article in 1855 Philadelphia warned of a farmer how had been the victim of a "gypsy swindle" by a "female fortune-teller" who needed a large sum of money to perfect a "charm" (all of the quotes are direct words from the article - which also calls the supposed spell "hocus pocus"). At face value it sounds like an average con but the entire story reads... suspiciously far-fetched. It's worth being extra critical as Roma were often blamed for every type of crime regardless of whether they committed it or not. Two specific examples I can think of regarding kidnapping claimed that "gypsies" had stolen the children, leading to local Roma being terrorized by locals and police; one case was a toddler that was later found to have fallen in a river and drowned, the other was a girl who was murdered by a white pedophile. (That story truly sticks out in my mind as the mother was reported to have said it was a relief that her daughter was merely raped and murdered, not still alive among the gypsies. Even if that quote was made up by the reporter, what the fuck.)

Fiction reinforces most of the long-standing prejudices and disregard for Romani people. The Hunchback of Notre Dame is a famous example of how prejudice only changes through the ages. In the original book, Esmeralda is a French (white) child who was stolen by the gypsies (another common racist stereotype for Roma, the origins of which can be summarized by "light-skinned Roma exist"); the Disney version "updated" the story to make Esmeralda a dark-skinned Romani woman fighting for her people... except the rest of the Roma in the film are caricatures of thieves and murderers, and it hyper-sexualized a woman from a group that is exceptionally vulnerable to sex trafficking in the real world.

Men are of course not totally insulated from the "evil magic" trope either, as exemplified by the forever-awful Stephen King story "Thinner." Unlike female examples of Esmeralda and Carmen, male examples are not sexualized and are more often linked to a criminal stereotype (again for Hunchback, think about Clopin and his whole "Court of Miracles" song).