When I was reading about the Portuguese first attempts to contact with the Ming Dynasty relationships between the two states eventually deteriorate due to rumors of the Portuguese capturing children as slaves. This led to a period of conflict between the two realms. Just a few decades later they began to partake in the enslavement of Japanese people as well, leading to their eventual expulsion from the Tokugawa Shogunate. Were the Portuguese just stupid and unable to understand how enslaving Chinese and Japanese people might ruin their prosperous Asian trade?
Before beginning, a note (more of a paragraph, or really multiple paragraphs, or even really really a chapter of a proper book written by a proper historian -- you can find such discussion elsewhere) should be made about the subject of 'slavery' and 'slave trade', that attached to it is a myriad of meaning and history that would serve us well to touch on before diving directly into the depths of facts and records.
To start with, slavery and the trading of slaves (more technically, western European involvement in them) is an area rich in research when considering those from Africa and the Americas, and almost entirely in contrast to it being non-existent in East Asia. This has changed in more recent years, but the body of literature has only just begun to bud from a small flower. This AH answer provides some good Japanese sources as well as two of the 'main' English secondary sources (one of which is a dive into Portuguese and Jesuit records). The AH answer you'll definitely want to read, the English sources you may want to read, and the Japanese if you are able then definitely do so :)
Secondly, there are deep cultural and linguistic ties to word 'slave' in the Western sphere that don't entirely align with their counterparts in the East. At least on the surface, slavery in Japan (which as an 'institution' had been established for hundreds of years prior to the Portuguese arrival) had more in common with indentured servitude or serfdom in the West, compared to the Atlantic slave trade -- see this AH answer for an overview, which is (again) recommended reading. The work by Nelson ("Slavery in Medieval Japan") is also a good start (and for now, end) point for this -- it's about 20 pages long and on JSTOR so if your school/university has access to it you should be able to read it.
Now that that's all out of the way, I'll try to tackle your question.
Slavery, in one form or another as recognisable to Europeans in the late Renaissance, existed in Kyushu (south-eastern Japan) for many centuries prior to Portuguese arrival.
In 1240, some three centuries before first contact with the West, a shogunal order was made prohibiting the buying and selling of human beings when not in times of hardship and famine (the topic of famine being a byproduct of politics is a whole different can of worms to open here):
Concerning the prohibition on the buying and selling of human beings: Over the generations the imperial court has issued innumerable edicts [prohibiting such trade], and these laws have been enforced by the shogunate. During the famine of the Kangi period [1229-1232], however, people [had no choice but to] sell their children and dispose of their servants (shoj?) in order to maintain their existence. Because [to hold to] the prohibition [in these circumstances] would conversely have caused the people to suffer, it was not enforced. Now, however, things have returned to their former state. Even so, we hear that assorted persons are flouting the law. This is an outrage. Let it be stopped immediately.
So it's clear that superficially, it was accepted that slavery was sometimes necessary, and only when times were good should it be outlawed.
When the next shogunate rolled around the corner, the practice continued, and then when all hell broke loose during the Sengoku period, seizing slaves as prisoners of war became custom (with regional variations, of course), and the same justification of hardship lead to many other individuals entering slavehood. When the Portuguese arrived, the supply of slaves in Kyushu was such that it was not difficult to participate in the process, and it was not immediately evident that this would be detrimental to their continued trading relationship.
A short final tangent here (although I'll defer to experts in their areas because this I know some of but not a lot).
First, the Portuguese colonial empire was not a single rigidly controlled organisation in the way that the Dutch India Companies were, and so orders by the King to prohibit slavery were not by nature easily enforced.
For instance, in 1571 (50 or so years prior to the Sakoku edicts) the then-King of Portugal issued laws prohibiting slavery under the pretext of it inhibiting the work of Jesuit missionaries in Kyushu (via the poor reputation it put on the Catholic missionaries). Nevertheless, the value of the slave trade in labour and sex accompanied with what I mentioned above meant that such a ban was almost never enforced.
On the other hand, Portuguese India (Goa) (who directly benefited from the slave trade) did not want such a ban at all, as slaves who were 'legally bought' should remain the property of their owners. This resulted in a situation where the Jesuit missionaries who were the foremost representatives of the West in Japan could excuse themselves by blaming the Portuguese merchants (and the Japanese who sold them the slaves), and it seems that in part this story was accepted by Japanese Christians.
So let's try and answer the questions from a summary of the above.
Were the Portuguese stupid and unable to calculate how this practice of slavery would ruin their trade? No, they were well aware, but not entirely empowered to stop it, and the rewards of the trade seemed to heavily outweigh the risk.
Did the Portuguese not worry that slave trade would ruin trade relations? No not really, it was more that they were worried it would ruin the ability of the Jesuits to convert Japanese locals to Catholicism. Trade relations did not factor in to this, and if anything, continued trading of Japanese slaves (and Korean slaves, by proxy of the Imjin war) would enhance trade relations with Portugal's existing colonies in East Asia.
[Not entirely a question you asked but part of the assumption which I want to challenge] Was slavery the primary reason for the expulsion of the Portuguese from Japan? No, although it certainly played some part in it. For more reading, I would say that any of the first four questions in the FAQ on the Edo Period (cheeky link) are a great summary for what did (and didn't) cause the Japanese to expel European influence (minus some Dutch ships). At the risk of going out of scope here, the main contributor was still Catholicism and religion, and not the slave trade alone.