Portugal had superior sea power and navigational capabilities; but Japanese society was highly militarized and they possessed large organized armed forces and strong weapons. Portugal's strengths were sufficient to subdue South American tribes and certain African coastal regions; but they did not have sufficient numbers to invade a more militarily advanced nation like Japan was at the time.

I doubt that there is any specific source I can point to that will explain this, however, the Portuguese - like other colonial powers of the 16th and 17th century - were really only able or interested in colonizing ports (that is outside the new world, and even then coastal settlement was still the extent of colonization). So, on the one hand you have this interest in coastal ports - both the product of a limited ability to project power and the perils of the interior to European immune systems.

On the other, Japan was a relatively consolidated polity. Prior to the wave of colonization of Africa and South Asia, most European colonization schemes were against small entities, weakly associated entities, or in regions that were sparsely populated. Japan had a form of centralized authority in the Shogun (and Emperor) and a feudal military structure that could have resisted. In addition, it was - when compared to other regions they had settled, a very populated place. Modern estimates of the population in 1600 range from about 15-17 million (see William Wayne Farris, "Japan's Medieval Population: Famine, Fertility, and Warfare in a Transformative Age"). Meanwhile, Brazil - Portugal's most populated colony - had a population of around 100,000 people (an estimate derived from multiple sources) which consisted largely of settlers and slaves.

Now the meat of the problem, it seems you are overestimating the Portuguese's ability to do something like this. You have to remember that there is about 38,000km (going around Africa and not straying too far from the coastal areas all along the way to take advantage of the Portuguese colonial ports) between Lisbon and, say a "relatively close" Japanese city, Nagasaki on Kyushu.

It would be a logistical feat to transport a large Portuguese army to Japan using only sailing ships and small ports along the way for resupply. You have to remember that even in the places where Iberians consolidated colonies most, in Latin America, Spanish conquistadors who first came to do so were relatively small bands. Cortes arrived with only 500 men to what is now Mexico - and remember this is over a distance of only about 10,000km (remember, about 38,000km was to get to Japan).