The common understanding is that prisoners in German concentration camps, especially Jews and Russians, were treated as expendible slave labor who were killed practically on a whim.
But, the Nazis were also very serious about order and discipline within their military. So, while certainly prisoners were executed regularly for little or no reason on authority from higher, I feel it would be unlikely that an ordinary low level guard would have been allowed to arbitrarily execute a prisoner on his own whim.
Would it have been allowed? Would there be more than a cursory investigation? What kind of punishment might such a guard face?
I'm trying to understand where exactly the line would have been drawn between the sanctioned killing of undesirables and maintaining discipline within their own ranks, and it just seems like allowing random executions by people without any military authority would undermine that discipline.
line would have been drawn between the sanctioned killing of undesirables
I would, as general advice, recommend to put quote marks around Nazi terminology such as "undesirables".
Anyways, the discipline in Concentration Camps among the guards was a social mechanism to justify atrocities against inmates. What I mean by that is that killings in the camps, when not done in a so-called selection, but rather by individual guards were usually done in a matter that can only be described as a pretense to discipline in order to give what occurred the appearance of having been legitimate in the eyes of the perpetrators and therefore easier to rationalize.
The most famous example of this is probably what has been described as "cap tossing". An internal regulation in the camps stated that all inmates had to wear caps as parts of their uniforms and not wearing caps would be punished by the guards severely. This regulation was regularly exploited by guards by taking an inmate's cap and tossing over the line that inmates were not allowed to cross in front of the fence or the guard posts. When the inmate would go to retrieve his cap, the guard would then shoot them in the back and record the death as "having been shot while attempting to escape". The guard would then receive a couple of days of payed holiday. In case the inmate would refuse to retrieve his cap, the guard would proceed to beat them until they either went to get the cap or died on the spot – a death that then would be recorded as having been the result of a justified punishment for refusing the guard's orders.
This is the most famous example of this mechanism because it's been documented over and over again through testimony both by former guards and former inmates in trials such as the Auschwitz Trial in 1963 and because it matches so many records we have for deaths in concentration camps (which were recorded like other deaths would have been, in part for legal purposes, in part because of the same mechanism of legitimizing the killing).
Similar mechanism can be observed basically from the beginning of the "professionalization" of the camp system in Dachau fro 1934 forward that developed the model on which all later camps were based on, including such things as the uniforms, the requirement that sleeping places had to be clean and a whole host of other things that were basically impossible under the conditions of the camp and could be used by guards to justify killing prisoners.
The theory of how this works was developed by Ernst Fraenkel in his book The Dual State, originally published in 1940/41. Fraenkel, a jurist by training, had been forced to flee Germany because of his Jewish heritage and socialist leanings and he was one of the first people trying to really grasp the political system of Nazis.
In his book Fraenkel posits that the Nazi system is marked by the simultaneous existence of a normative and prerogative state. The normative state as defined by Fraenkel is what we generally would classify as a state under the rule of law, meaning that like we are used to, every action taken by the state is based on laws (in democracy we generally accept that the state can only do what is explicitly conferred to him as part of the legal framework). The prerogative state on the other hand is the state that takes actions not based on law but based on situational-political expediencies in line with ideological goals.
Fraenkel points out that things like private law, relations of property, torts, contracts and so forth still functioned like they had before in the Third Reich with laws regulating them. with courts deciding disputes and with the state administrations decisions still being binding – except where it concerned the Jews and thus fell under the prerogative states mandate. While there was a slew of anti-Jewish legislation, the way state agencies, courts, and so forth acted towards Jews was not guided by legal norms very much in place in other cases but solely by political and situational expediency.
While in many situations the state continues to function as it has previously functioned, in certain areas, agencies like the Gestapo of the Camp SS and others are completely freed from legal and other restraints in their actions but in order to uphold its own legitimacy in the eyes of both those not involved, and even more importantly in the eyes of those involved, the perpetrators, it had to develop an image of actually belonging to the normative state. The motions of going through things like tossing the caps was not just a cruelty on part of the guards, it was an essential mechanism so that the guards could justify their behavior and actions under the moniker of "doing their duty" or "doing waht is required" and staying within rules layed out by the system.
Fraenkel puts it the way that the prerogative state is the superior of the two because it can supersede the normative state where it's wielders feel appropriate. At the same time the prerogative state is dependent upon the normative state, not just because without a normative state a capitalist economy would simply cease to function (to be able to gauge the consequences of one's actions and enjoy legal protection for a contract e.g. is essential to how capitalist economies function) but also to elicit the collaboration of the traditional state authorities in the ideological mission of the new state and for those involved to feel justified about what they are doing.
The important lesson from this is that states and societies function en large via fiat: Once citizens and bureaucrats, judges, police men and so forth start losing faith in the state's ability to function, a state effectively stops to function because it is impossible to sustain a social system in the long run without the people's faith in it. Even massive violence requires those who exact that violence in the name of the state. And so, the normative state is important because to the privileged group that enjoys its application to them it is essential to maintain the fiat in the functioning of the system as a whole. It is important to the system as a whole to still execute laws in normative way where the privileged majority is concerned in order to maintain the faith in the system necessary to institute the prerogative state measures against the marginalized and oppressed groups.
Sources:
Nicholaus Wachsmann: KL. A History of the Concentration Camps.
Ernst Fraenkel: The Dual State.
Wolfgang Sofsky: Die Ordnung des Terrors: Das Konzentrationslager.