I never understood that, why did they call them southern barbarians? Relative to Japan, their countries were on the west, and they often came from the direction of India and China, did they not? Wouldn't it make more sense for them to be either Western Barbarians or Northern Barbarians?
Nanban didn't start out meaning Europeans. It actually originates as a Chinese term Nanman, meaning "Southern Barbarians" and was used to refer to non-Chinese ethnic groups to the south. That term was widened to South-East Asia and India, and adopted by the Japanese with the pronunciation nanban.
Olof G. Lidin writes about the earliest recorded Japanese use of the word:
In parenthesis it should in this context be noticed that the two terms, nanban, 'Southern Barbarian', and nanbansen, 'Southern Barbarian ship', were used in Chinese parlance before the Europeans arrived.They referred to ships and people arriving from the South generally. Nanbansen coming from Southeast Asia are mentioned from early times with reference to lands south of Kyushu. The arrivals of a nanbansen from Sumatra as early as 1408 and of a nanbansen from Java a year later are recorded in the annals.
When the Portuguese showed up in Japan, they arrived from the south, as part of the same South-eastern Asian trade networks. They therefore fell under the umbrella word nanban.
One early document actually quotes a Chinese translator identifying the Portuguese to his Japanese audience as seinanban ie. south-western barbarians, because they come from so far west, but nanban is the general term.
In time nanban shifts to just mean the Europeans, and I couldn't find any detailed analysis of how it changed, just explanations that it had shifted by the end of the sixteenth century. The European presence in Japan during the late sixteenth century certainly made a big impression, enough to take over the word for themselves.
In order to avoid giving a superficial answer to yours question, I'll preface the answer to your question with some context. The etymology of the word 南蛮 (nanban; "southern barbarian") took its roots from the Chinese word 南蠻 (nanman; "southern barbarian"). While both share a similar meaning, the Chinese word was a catchall term used to refer to the peoples of Southern China unlike the Japanese word which came to refer to Portuguese and Dutch traders and (later) all other white races (including colonial nations).
Despite it's position as a fragmented island nation, Japan's trade with foreign nations was rather limited; the bulk of its trade came from dealings with Imperial China and what is now called Korea. In fact, most of its maritime tradition was rooted in piracy by Wakō pirates (倭寇 Chinese: Wōkòu Japanese: Wakō; "Japanese bandits" or "dwarf bandits") who would raid the coasts of Korea all the way to Hainan in Southern China. As such, Japanese contact with foreign cultures excluding Korea and China was extremely limited due to its isolated geographic position among developed societies.
Now on to directly answering your question. Japan's first contact with white foreign traders was in the Satsuma domain (what is now southern Kyūshū). These initial encounters created a culture shock that the Japanese were unaccustomed to. While the Japanese had been trading with Chinese and Korean traders for centuries, they initially found it very difficult to interact with the new arrivals. This is because while regions of Japan, China and Korea have different languages, to some extent they all used the Chinese writing script for all official documents. Until the arrival of the Jesuits (and later Dutch trade missions) to the region, this communication problem would persist. This lack of ability to communicate with traders in combination with their strikingly different facial features and skin colour warranted its own categorisation. The Japanese adapted the Chinese word Nanman (mentioned at the start of this post) to refer to a completely different set of peoples.
So why Southern? Simply because Japanese geographical knowledge of the world was limited to East Asia. Portuguese and dutch traders would arrive to Japan via the Southern island of Kyūshū using the Cape of Good Hope trade route discovered by Vasco da Gama only 30 years earlier. From the Japanese perspective, these people were coming to Japan from the South. It wouldn't be until much later with the arrival of colonial trade missions from Mexico that Western traders would arrive to Japan via other ports, eventually leading the Shogunate to restrict trade to certain ports. However, by the time that the Japanese came to understand the geographical position of Western nations, the term Nanban was already widespread enough that it became the defacto word to refer to Western traders. While the term isn't really used aside from in its historical context, it lingers on in modern culture with the dish "Chikin Nanban" (Southern Barbarian/Portuguese style chicken), a Kyūshū region speciality supposedly introduced to the region by the Portuguese.
Sources:
The making of modern Japan Marius B. Jansen Japan and the Shackles of the Past R. Taggart Murphy In search of the roots of Southern Barbarian style cuisine Makiko Kataki