Part 1

The key to answering this question is that in your assumption about rationality and the German war effort, you are not taking into account that the war the Germans fought was a war with declared racial goals from its very beginning. Following Clausewitz dictum of war being the extension of politics by other means, the politics of the historical actors were those of a campaign of racial annihilation and due to this, to the historical actors, the actions of the Holocaust were rational and the resources directed towards it in a rational manner were also spend on an aim they considered worthwhile.

Again, in line with the above cited dictum by Clausewitz, we tend to see modern war as a struggle for political and economic hegemony over certain areas and between states. This is also true for the war the Nazi state fought but it was also more than that. For Hitler and the leadership of the Third Reich, the war they started by invading Poland in 1939 was also always a racial war. A racial war in the sense that for Hitler history was ruled by the law of race struggle and the purest expression of that struggle was war. He and the rest of the Nazi leadership sought to fight the war to end all wars in Europe resulting in the total political and racial hegemony of the Aryan race.

How these principles shaped how they fought this war was apparent from 1939 and only increased over time. When the Wehrmacht marched into Poland in 1939, she did so not only with less regard for civilian life than had been displayed at least in certain areas in WWI (the bombing of Warsaw comes to mind) but also with the SS Einsatzgruppen closely on their trail. The Einsatzgruppen in Poland were charged with conducting rear security for the Wehrmacht. How they understood this task gives us an impression of what the aims of this war were. In between the invasion of Poland in September 1939 and December of the same year, the Einsatzgruppen murdered 65.000 people. Their victims were the Polish intelligentsia, i.e. priests, politicians, intellectuals, authors etc. as well as politically active Jews and some Roma communities. The purpose of this killing spree was to physically wipe out the people most likely to lead the Polish resistance against the occupation as well as to kill any elites from which the notion of Polish nation could persist. The Poles were to serve the Germans as subhuman slaves. They had no need for any kind of political or intellectual elites and in order to prepare them for their serfdom, their leaders and intellectuals had to be killed. The war in Poland was from its very beginning fought as a war of racial dominance and the campaign of murder by the Einsatzgruppen was seen as a first step of racial consolidation of Poland.

This is important to mention because similarly to the Polish intelligentsia, the Jews in the eyes of the political and military leadership of the Third Reich always represented a security risk. Jews were seen as the puppet masters behind Communism and Partisan resistance. "Where the Jew is, is the Partisan and where there is the Partisan, there is the Jew" ran the Wehrmacht moniker. This thinking becomes apparent in Serbia in 1941 when the Wehrmacht encounters serious Partisan resistance due to the communist and nationalist uprising against the occupation. The immediate response of the Wehrmacht aside escalating violence against civilians is to write to Berlin to deport the male Jews of Serbia to Poland because in their mind, it's these people who are responsible for the uprising. When that doesn't work out for several reasons, the Wehrmacht commander, Franz Böhme, orders all male Jews shot as part of the anti-Partisan campaign.

This example serves to illustrate the for the Nazi and military leadership, racial ideological thinking was so deeply ingrained in their idea of how to conduct this war that the Holocaust as in the systematic murder of all of Europe's Jews became an integral part of the war comparable in its importance to, let's say, building tanks. The same way they thought they could not conduct their war without tanks, they thought they conduct their war without killing Jews once they started resp. this was also a factor in what lead them to start the killing in the first place as I discuss here and here.

With this underlying mindset in mind, the organization of the Holocaust and the decision for several other atrocities taken where designed in a fashion that assisted the German war effort further than just satisfying their idea of security through killing Jews.

The Holocaust was essential a "for-profit-venture" that in its execution was designed to be an integral part of the German war effort. By applying complicated schemes described in this answer, it not only paid for itself, it also helped fund the German war effort. Stolen Jewish assets worth millions; the fact that Jews paid the Reichsbahn for their train tickets, that certain satellite states like Croatia in essence paid the Germans to deport their Jews; the exploitation of Concentration Camp and Ghetto prisoners for forced labor; gold teeth from the victims and their hair being sold for mattresses and U-boat boots; the relatively low cost of killing operations by using tank engines to gas people – all this lead to the Holocaust contributing more to the war effort than diverting away from it.

Also, the Holocaust didn't bind a lot of man power. The Einsatzgruppen in the Soviet Union who murdered 500.000 until the end of 1941 had three thousand members. The total staff for the Operation Reinhard Camp in which 1,5 million people were murdered were 500 people. The number of concentration camp personnel never comprised more than 10.000 people total and that includes secretaries, people unfit for active duty in the Wehrmacht, administrative personnel, and non-German personnel, which the Nazis would not have used in active fighting.

Additionally, the Concentration Camp provided much needed cheap man power without which the Nazis would have never been able to fight as long as they did. The German economy lacked man power even in the 1930s. When the war started, the Nazi immediately began basically kidnapping people from their homes all over Europe to work in Germany. By August 1944 25% of the total labor force in Germany were forced laborers; civilian, POWs and from the camps. That's more than seven million people or the population total of Austria back then. All in all it is estimated that 12 million people worked as forced laborers for the Germans during the war. IG Farben and various German armament undertakings including reddit's much loved V2 rockets could not have done without the labor force of concentration camp inmates, which were needed just to make up all the people drafted to the Wehrmacht and to satisfy the ever rising man power need.

Furthermore, in their economic planning, murder was essential for the planning of the German food economy in WWII. The leadership of the Third Reich remembered the end of WWI well when the hunger winter of 1917 had been a decisive factor in the revolts and mutinies that ended the war. For them, keeping the Germans fed was a top priority target. Feeding millions of Jews or Soviet citizens for that matter was not high on their priority list.

Historian Christian Gerlach asserts that the reason why Aktion Reinhard, the killing of the majority of Poland's Jews, started in 1942 rather than later when it was originally planned, had to do with the German food situation. Unwilling to lower rations for the German population, the Nazis reduced the need for food supply by killing a considerable amount of Jews. You don't have to feed dead people.

This argument is backed up when looking at the Nazi plans for the invasion of the Soviet Union or the policy in Greece. In the USSR the Nazi Generalplan Ost foresaw at least 3 million people, mostly the urban population, starving in the first year of the war so that the Wehrmacht and the German population could be fed. These plans went even further into letting 27 million people starve but could not be exacted. However, the group that also fell victim to this plan were the Soviet POWs. Of the 3 million POWs captured in 1941, half starved until the end of the year. All in all 3 million Soviet POWs died so that the Nazi leadership could feed the Germans. A similar situation developed in Greece, where the Nazis produced a man made famine by exporting all food stuff to Germany. 300.000 Greeks starved to death during Nazi occupation so that the German population could be the second best fed of the whole war after the US population.

The essence or TL;DR answer to this question is that for the war the Nazis planned to fight, especially the kind of war they planned in the Soviet Union and they had to plan after the end of 1941, mass murder was essential tool that was applied in a horrifyingly rational manner. The death of millions of people became essential part of the strategy of war and war economy rather than something seen as detrimental or divorced from the war. Without the Holocaust and other murderous policies, Nazi Germany would probably not have been able to carrying on fighting as long as it did, especially relating to issues of funding the war effort and supplying the German populace with food.