When you got your vaccine, they gave you an inoculation with vaccinia virus. I'm not well-educated about vaccinia but I believe that, at least the strain we use for vaccines, can only be spread by skin-to-skin contact. It doesn't usually result in a full infection, anyway, because the vaccine virus is attenuated (weakened).

Natural smallpox virus (variola, not vaccinia) could spread through the air. Smallpox is an airborne infection, as the 1970s Meschede outbreak showed: it spread throughout a hospital, up floors during quarantine, to people who had never seen the index patient's face. So it would've been readily spreading even without touch, possibly even before people developed the telltale pustules. In addition to the airborne route it would've been spread on fomites, objects touched by a person who was infected. Like clothes and, yes, blankets. In this way an infection easily spreads within a household and, in time, to the rest of the community.

It was thought of as a disease of "filth" for a long time because it spread so readily in the slums, where people living in poverty were in close contact with one another. But the airborne spread ensured it would break out of the impoverished streets into the homes of the wealthy as well.