"Fun fact", the concept of crimes against humanity still isn't well defined in international law and any discussion of that falls under the 20 years rule of this sub because there still hasn't been broad legal agreement.

EDIT: Please see /u/RobertoBolano's comment below on the Rome Statute which came into force in 2002 and explicitly defines crimes against humanity and governs the court's ability to prosecute individuals for those crimes. Note that China & India (amongst others) have not yet ratified the statute.

However, that's not what your question was about. As /u/FatherStorm has already pointed out, war crimes were prosecutable at the time of the Nuremberg trials. What was not explicitly prosecutable were crimes committed by a government against its own citizens during a time of war.

Now I'm not a lawyer and I'm getting my information primarily from reading on the Nuremberg trials and from having visited the courtroom's exhibits in the city so hopefully someone can correct me if I make mistakes.

The key thing was that a charter (called the London Charter but it came to be referred to as the Nuremberg Charter) was established to determine the code of conduct of the Nuremberg trials. In particular, what you're looking for is Article 6 which deals with jurisdiction and how the crimes would be treated.

In particular, section 6.c states:

Crimes against humanity: namely, murder, extermination, enslavement, deportation, and other inhumane acts committed against any civilian population, before or during the war, or persecutions on political, racial or religious grounds in execution of or in connection with any crime within the jurisdiction of the Tribunal, whether or not in violation of the domestic law of the country where perpetrated.

However, from notes and documents relating to the Nuremberg Trials, the Tribunal itself limited its own powers:

With regard to crimes against humanity, there is no doubt whatever that political opponents were murdered in Germany before the war, and that many of them were kept in concentration camps in circumstances of great horror and cruelty. The policy of terror was certainly carried out on a vast scale, and in many cases was organised and systematic. The policy of persecution, repression and murder of civilians in Germany before the war of 1939, who were likely to be hostile to the Government, was most ruthlessly carried out. The persecution of Jews during the same period is established beyond all doubt. To constitute crimes against humanity, the acts relied on before the outbreak of war must have been in execution of, or in connection with, any crime within the jurisdiction of the Tribunal. The Tribunal is of the opinion that revolting and horrible as many of these crimes were, it has not been satisfactorily proved that they were done in execution of, or in connection with, any such crime. The Tribunal therefore cannot make a general declaration that the acts before 1939 were crimes against humanity within the meaning of the Charter, but from the beginning of the war in 1939 war crimes were committed on a vast scale, which were also crimes against humanity; and insofar as the inhumane acts charged in the Indictment, and committed after the beginning of the war, did not constitute war crimes, they were all committed in execution of, or in connection with, the aggressive war, and therefore constituted crimes against humanity.

(my emphasis)

Also, the Tribunal did consider relevant international law (ibid):

The Tribunal is of course bound by the Charter, in the definition which it gives both of war crimes and crimes against humanity. With respect to war crimes, however, as has already been pointed out, the crimes defined by Article 6, section (b), of the Charter were already recognised as war crimes under international law. They were covered by Articles 46, 50, 52, and 56 of the Hague Convention of 1907, and Articles 2, 3, 4, 46 and 51 of the Geneva Convention of 1929. That violations of these provisions constituted crimes for which the guilty individuals were punishable is too well settled to admit of argument.

(my emphasis again)

So, to summarise, the Tribunal defined crimes against humanity in its charter established to conduct the Nuremberg Trials but then further defined them solely in conjunction with war crimes which were already well defined under international law. In other words, crimes against humanity were not separable and independent from the crime of unjust war but were subsumed within that category such that it was only permissible to adjudge a defendent guilty of crimes against humanity if it were demonstrated that they had committed a war crime in the commission of the crime against humanity.

So, the Tribunal took the stance that there was no retroactive application of laws because such laws were well established prior to the inception of hostilities on Sep 1, 1939.

Finally though, the concept of a "crime against humanity" did pre-date the Nuremberg trials. It came up during my research for this post that a joint telegram was issued by France, Great Britain and Russia in 1915 addressed to the Ottoman empire which stated:

For about a month the Kurd and Turkish populations of Armenia has been massacring Armenians with the connivance and often assistance of Ottoman authorities. Such massacres took place in middle April (new style) at Erzerum, Dertchun, Eguine, Akn, Bitlis, Mush, Sassun, Zeitun, and throughout Cilicia. Inhabitants of about one hundred villages near Van were all murdered. In that city Armenian quarter is besieged by Kurds. At the same time in Constantinople Ottoman Government ill-treats inoffensive Armenian population. In view of those new crimes of Turkey against humanity and civilization, the Allied governments announce publicly to the Sublime-Porte that they will hold personally responsible [for] these crimes all members of the Ottoman government and those of their agents who are implicated in such massacres.

(my emphasis) - also note that the telegram was sent by the USA Dept of State and thus carried the endorsement of the government of the USA at the time.

It is interesting to note that the telegram does explicitly state that members of the administration would be personally held responsible for the genocidal actions, a point which was later echoed in the Nuremberg Trials.