Because he did not become Prime Minister until mid October of 1941, having been Army Minister before that. And having come to power with the express purpose of winning the War in the Pacific there was little reason to travel halfway around the world to allies that could provide little direct support.

Same with Konoe his predecessor. He was not present at the signing of the Tripartite Pact, that is the entire point of ambassadors. It should also be noted that Mussolini was not at the signing either. Though of course he was personally with Hitler on multiple occasions.