I am not an early modernist, so I cannot go into too much detail here, but your categories of options are entirely wrong. Neither "experiment" and "scam" were the concerns for the time, because the modern language of science (i.e. "experiments," which is a word that barely even existed in its modern form at the time, and is not at all established as a way of generating knowledge about the world) and the fear that learned people might be leading them on are both later sensibilities.

The real debate would be whether or not alchemy constituted a form of heresy or religious apostasy, whether it was just an attempt to create material wealth (i.e. turning lead in gold), or whether it did somehow give insight into the mind of God, eternal life, or other outcomes that may or may not be acceptable with the religious faith you and your country ascribed to (which matters a lot during these years of Reformation, Counter-Reformation, and religious revolution).

So I can't answer the question (I know not a thing about Rudolf II), but I can, I hope, just point out that the world of the 15th and 16th centuries had very different worries and concerns about this kind of thing. By the 18th century, the idea that alchemy was at best proto-science and at worst a scam was common, but that was not at all how the practitioners — or patrons — saw the stakes at the time.

On alchemy in general, the work of Laurence Principe is supposed to be very good. I have looked at some of this literature but it has been a long time.