You could probably argue pretty convincingly that the Holocaust was a crime under the customary international law of the day, because I think genocide or mass murder at that scale would be considered a peremptory norm.

But in a more strictly legalistic sense — i.e. in terms of codified international law — the Holocaust directly prompted the creation of the Convention for the Prevention and Punishment of the Crime of Genocide (CPPCG or Genocide Convention), Article 2 of which created an explicit definition of genocide in international law:

In the present Convention, genocide means any of the following acts committed with intent to destroy, in whole or in part, a national, ethnical, racial or religious group, as such:

(a) Killing members of the group;

(b) Causing serious bodily or mental harm to members of the group;

(c) Deliberately inflicting on the group conditions of life calculated to bring about its physical destruction in whole or in part;

(d) Imposing measures intended to prevent births within the group;

(e) Forcibly transferring children of the group to another group.

So if it wasn't a crime before the Second World War, it was made one very soon after (and as a direct result of the Holocaust.) That transition from customary to treaty law is important in international law, because customary international law is notoriously difficult to prosecute or arbitrate. Customary law is by its nature (i.e. "derived from custom") open to subjective interpretation and manipulation.

It's also worth noting that the two world wars — and particularly the second — played an enormous role in the formation of modern international law. The idea of international law is a 19th century conception, and the League of Nations was in large part built and dependent on the desire for international law to become a universally-recognised concept.

But the failure of the League was pretty absolute; the institutions that persist today — however toothless or ineffective they may seem at times — are those of the United Nations, not the League. The legal branch of the League, the Permanent Court of International Justice, was dissolved in 1945 and replaced by the International Court of Justice. Essentially, the First World War wasn't shocking enough to force the world into consensus around the idea of international law and formal international governmental organisations; the Second certainly was.

I don't know much about German jurisprudence under the Nazis, but given the legalistic nature of that regime I don't doubt they could have found a legal rationale — however spurious — for their actions.

Also: you have to be careful with the term 'crime against humanity', because it has a specific meaning in international law and is distinct from the crime of genocide.

That was a serious concern for prosecutors in the lead-up to the postwar trials. For instance, no one was prosecuted for stripping German Jews of their citizenship in the Nuremberg Laws, because it was a legal process and the prevailing view was that nations were more or less free to do what they wanted with their own people within their own borders. That's why if you look at specific trials and cases, the prosecution tended to focus on crimes committed by Germans against foreigners in occupied countries or foreigners brought to Germany.

The legal basis for this was The Hague Conventions of 1899 and 1907, to which Germany was a signatory (though Allied prosecutors also argued that since so many countries were signatories of The Hague a Conventions, all nations were sufficiently aware of their contents that they should abide by them as well). The 1899 convention had specific clauses regarding the conduct of occupying armies and the rights of occupied peoples, which Germany was clearly in violation of while implementing the Holocaust.

Finally, German civil law remained in place during the Third Reich, including provisions agains murder and rape, so individuals could still be prosecuted for these crimes.