I've heard rumours that Princess Anna was supposed to be in charge, but then she ran away for some reason. But even so, our bishop was still around. Hell, even the Duke of Weselton was there, isn't his duchy a close trading partner of our kingdom and thus would be a more reliable regent than than some idle prince from the south? What does our constitution say about who is supposed to be in charge when a monarch runs away?
As always with matters pertaining to royalty, the sequence of events between Hans being appointed as temporary head of state and Princess Anna leaving the Royal Palace is not entirely certain. As you rightly point out, the choice of Prince Hans of the Southern Isles as temporary monarch is bizarre; a lowly prince, last in line and without any previous experience in leading a kingdom. Some of his defenders, in particular these last few years, have pointed to his career within the admiralty of the Southern Isles as proof that he was more than capable of ruling a kingdom for a short time while his detractors point to the foolish expedition to the North Mountain where Queen Elsa was supposed to be - effectively abandoning the kingdom without appointing anyone in his stead.
To understand the why, we have to understand the dynamic between Anna and Hans at this vital moment in Arendelle history. According to Eugene Fitzherbert's memoirs, he overheard an argument between Queen Elsa and Princess Anna just minutes before her disappearance. Judging from what he overheard, it appeared that Princess Anna had accepted the young prince's marriage proposal to Queen Elsa's dismay who was now urging her sister to break up the engagement. Several other eyewitnesses more or less agree to this point, having heard Princess Anna openly and loudly protesting against her older sister's wishes. Less than 10 minutes later, Queen Elsa had not only shown that she was in posession of supernatural powers but had also fled the scene. There are some historians who speculate that Queen Elsa's display was triggered by the anger she felt at her sister's choice, pointing perhaps to a possible previous love affair between Queen Elsa and Prince Hans.
As Princess Anna and Prince Hans ran after Queen Elsa to no avail, since Queen Elsa had crossed the grand lake in front of the Royal Palace with the use of her powers, Princess Anna was heard giving the order to her servants to bring her her horse. The man who did, the Arendelle royal overseer Kai, was the only man besides Prince Hans who was present when Prince Hans was appointed temporary ruler of Arendelle by Princess Anna before her public declaration that he had been. In an unaired interview recorded before his death and recently aired as part of the BBC World Service programme Witness, Kai says how he heard Princess Anna say that she needed Prince Hans to stay in Arendelle. There are few historians that wouldn't interpret this as Princess Anna making a swift decision in the moment based entirely on her personal feelings towards Prince Hans. One historian in particular (Kim Jørgensen in his monumental The Rise and Fall of Prince Hans of the Southern Isles) has pointed out the fact that the Duke of Weselton was in such shock at this time that he was not fit to reign. There are also signs that the Duke suffered possible injuries at the hands of Queen Elsa during her first display of her powers.
The pre-Queen Elsa reign constitution which had not been altered during the chaotic time after the death of her parents, had allowed this to be possible during an emergency period. Queen Elsa, who formed the Stortinget (Arendelle parliament) during her reign, brought a new constitution which gave the Stortinget exclusive powers to choose a temporary ruler in case of emergencies.
To sum things up, Princess Anna's personal feelings and engagement to the Prince heavily affected her decision to appoint Prince Hans as temporary ruler of Arendelle without considering his merits.