I can't comment upon the Americans, but for the British army the change arguably came with the introduction of the L85 rifle (if I stick to the letter of your question). Previously, officers had carried sub-machine guns, or before this pistols. The reason why was due to the fact that they were not generally called upon to join the firing line, only in extremis rushing in to lead their men by personal example. Their job is to develop and order a plan, co-ordinate the wider battle, communicate with the superiors and their supporting arms, not to act as a rifleman, blazing away at the enemy, or getting far forward where they are unnecessary expose to enemy fire, preventing them from accomplishing the missions which they were trained and paid for. Giving them weapons with limited range, magazine or firepower would reinforce this, and consequently pistols of SMGs were popular choices.

In terms of the shift from pistols to SMGs, it was during the second world war that it became common practice for British empire and commonwealth subalterns to exchange their pistols for their signaller's submachine gun. This gave them greater firepower, while lightening the load for their signallers. Later, the poor quality of the Stirling SMG led to most men dumping this weapon in favour of the SLR (FN FAL) accepting its greater weight for uniformity, reliability and hitting power, this was especially prevalent in the Falkland's were these weapons (and snipers) were ubiquitous.