In many cases, German paramilitary groups were maintained by conservatives in the German government/military to keep order and engage in street fights with radical socialist groups in the general disorder that occurred in 1919. In most cases, these paramilitary groups were composed of former soldiers who had fought in the First World War and found themselves out of the army following Versailles. The treaty severely reduced the size of the German military from many millions of men to, as you note, less than one million. During this period of upheaval, the paramilitary groups (such as the Freikorps or Stahlhelm) were organized by private interests as militia or unofficially by the government/military as a reserve force.

They would have armed themselves accordingly depending on how they were organized. Many, constituting the Schwarze Reichswehr, were armed unofficially by the German government - they would have been armed from the weapons that the German army and government already controlled. Even though the army's "official" size had diminished, this did not mean that the massive armaments the German Empire had stockpiled were gone, nor did it mean that the conservative leaders of Germany had any intention of following the Treaty of Versailles.

Otherwise, many Germans would have owned hunting or sport rifles, so finding the basic armaments for militia would not have been too difficult. Schuetzenverein (marksmen's clubs) had existed for several decades already in Germany, and some men in paramilitary groups would have been members of a club.

Sources:

1. https://www.britannica.com/topic/Freikorps

2. Hagen, William. German History in Modern Times: Four Lives of the Nation. New York: Cambridge University Press, 2012. Page 250.