Specifically, the smallpox vaccine existed before the development and acceptance of the germ theory. So how did doctors think the smallpox vaccine worked?
In my limited knowledge, at the time doctors accepted the "miasma" theory of bad air. Did doctors think smallpox vaccine counteracted the bad air or something?
Did Chinese doctors, who made the first smallpox vaccine, have their own theory?
Joseph Needham reported first reported the apparent correspondence of the inoculation of smallpox (distinctly different from a vaccine as inoculation requires actively infecting someone with it, whereas a vaccine uses inactive or severely weakened infectious matter) in the 1500s in China, but attributed it to some kind of secret alchemical process that potentially dated to the 1000s.
Needham postulated that a certain aspect of smallpox was recognized by medical thinkers-namely, people who had survived smallpox once were not likely to contract it again, or if they did contract it it was much milder than that of a previously uninfected person. The thought was that the smallpox that had infected these survivors was somehow weaker than the pox that killed its victims. Therefore, since smallpox survivors did not contract the smallpox again, and smallpox survivors had a weaker version of smallpox, it boils down a rather simple problem of logic that you could avoid the lethal version of smallpox by contracting the weaker version. The scabs left over from the pox were a clear indicator of said smallpox, so scraping the scabs and ingesting them in various means-for instance inhalation-were a natural vector for this variolation.
Over time, by preparing the scabs and/or other infection vectors, they were able to further weaken the virus, such as by letting the scabs dry, by cooking them, and so on. This thus allowed them to reduce the risk of actually contracting full blown smallpox, although it is unlikely that the germ theory basis of what they were doing occurred to them.
We don't need to wonder, because Edward Jenner, the discoverer of vaccination (which pre-Pasteur referred only to using cowpox to protect against smallpox -- vaca is the root word, from cow) describes it in his third paper on the subject "A CONTINUATION OF FACTS AND OBSERVATIONS RELATIVE TO THE VARIOUS VACCINES, OR COW-POX" from 1800 (you can read here: https://biotech.law.lsu.edu/cphl/history/articles/jenner.htm#paperIII).
In it he discusses the best ways to preserve cowpox in order to transport it; at this point, the standard was chain vaccination. In the Balmis-Salvany expedition, they literally innoculated cowpox from arm to arm across the long sea voyage.
So in his third point, he intimates how vaccination works --
Thirdly. That the first formed virus, or what constitutes the true cow-pox pustule, invariably possesses the power I have ascribed to it, namely, that of affecting the constitution with a specific disease, is a truth that no subsequent occurrence has yet led me to doubt. But as I am now endeavouring to guard the public as much as possible against erroneous conclusions, I shall observe that when this pustule has degenerated into an ulcer (to which state it is often disposed to pass unless timely checked), I suspect that matter possessing very different properties may sooner or later be produced; and although it may have passed that stage wherein the specific properties of the matter secreted are no longer present in it, yet when applied to a sore (as in the casual way) it might dispose that sore to ulcerate, and from its irritation the system would probably become affected; and thus, by assuming some of its strongest characters, it would imitate the genuine cow-pox.
What?! A virus?! How could that be?
So virus prior to germ theory referred to a poison substance, from contagion theory (as opposed to miasma). This dates all the way back to Fracastoro in the 16th century, and refers to the idea that disease is transmitted via "germs" which are chemical substances that can trasmit from either person to person via contact, or by intermediates called fomites. So Jenner felt that vaccination worked because it was a virus with similar qualities to smallpox -- not an explicit understanding of immunity (that would take almost a century more), but given the long-held obersavtion that smallpox infetion protected against future infection, pretty much in line with our current understanding. The idea of "analogy" is one of the oldest in medicine, and dates back to the ancients. And in fact, vaccination would spurn some disasterous copy-cat attempts on a similar line, especially "syphilization" -- which mostly just gave a lot of patients syphilis.