I can't speak for witch trials across all of Europe and America, but a good case study of early modern witch trials might be the North-Berwick trials in Scotland. There were examinations and trials in 1590-91 and then another outbreak of prosecutions in 1597. During this time the king of Scotland, James VI, who would go onto to rule all of Britain, took great interest and leadership in investigating witchcraft. James wrote his own epistemological work on the nature of witchcraft known as his 'Demonology'. His studies were largely based off of his own experience in witch trials and the previous works of demonologists.

'demonologists are likely to bring certain theoretical presuppositions and expectations to bear, and these are likely to affect and shape the testimonies of the accused and the records of those testimonies. Those records, or memories of them, are in turn fed (along with other materials) into the demonological works which these men write.'¹

An interesting example of this working in practise was the theory that the devil, being unable to 'create', would take on a non-physical form when dealing with witches. Not as a ghost but rather like a being whose form is made solid by its lack of matter. As such the devil's body would be extremely cold to the touch. This idea seems to have influenced testimonies. Women who confessed to sexual intercourse with the devil (which theoretically was always included as a way to seal the pact between the witch and the devil) would also be recorded to have said that they did not gain much satisfaction from the experience as the devil's body felt deathly cold.

In the early modern period the study of witchcraft became far more intellectual in nature. But works like demonologies had many more purposes and prejudices to be purely critical works, they were written as to warn the reader away from the temptation of witchcraft and could be argued to have been written around witchcraft trials in order to justify conviction rather than as a method of deliberating fairly on trials.

To answer your question, yes, epistemological works on the nature of witchcraft had increasing influence over the handling of witch trials. However, I would doubt to what extent these works challenged current beliefs and practises than rather justify them.

Source used: Witchcraft in early modern Scotland: James VI's demonology and the North Berwick witches. Lawrence Normand and Gareth Roberts. Liverpool university press.

This text contains both a full copy of James VI's demonology, other contemporary sources and historians commentary on the topic I've been lurking on Reddit for a while and made an account just to make this comment