James Fujitani's article "The Ming Rejection of the Portuguese Embassy of 1517: A Reassessment" answers this question, so if you have access you should give it a read. I will give a summary here.
Despite the significance of the event to our modern eyes, it being the first time an European nation-state made official contact with China in the early modern era, China at the time did not treat the Portuguese embassy as a historic event. Instead, the Chinese treated it like another state from Southeast Asia trying to present tribute, which was a relatively common occurrence that the Chinese had a set of rules for. Also, the Ming dynasty had traditionally banned private maritime trade, which was only relaxed in 1509. At the time of the Portuguese arrival, there was still a lot of conservative forces hoping to restore the ban.
As with a lot of things, it's usually a combination of a variety of reasons that contributed to the final decision, so it is hard to say which was the actual reason China rejected the embassy. It is notable that Chinese and Portuguese sources mostly agree on a set of reasons, though they can be a bit simplistic. So let's discuss some of these reasons:
The Portuguese took over Melaka, a Chinese tributary state in Malaysia, in 1511. Its deposed king went to the Chinese court denouncing the invaders and requested China's help in taking back his kingdom. The Portuguese explained that they only acted in the best interests of the Melakan Chinese, who complained of the king's tyranny. The Portuguese must have thought that this explanation would score points in the eyes of the Chinese court, but the Ming dynasty had strict laws forbidding the Chinese people from going out to sea - essentially, the Portuguese had just told the Chinese that they sided with criminals who broke the law and fled overseas!
The Portuguese were unfamiliar with Chinese diplomatic customs and committed a series of faux pas. Other than the letter mentioned by OP, the Portuguese also fired their guns in salute when they arrived at the port of Canton (Guangzhou). The Portuguese said they saw Chinese merchants doing the same thing in Melaka, an explanation which, referring to the point above, must have annoyed the Chinese officials again. Despite this, the embassy actually had a positive reception! The interpreter that the Portuguese hired knew where to grease the wheels and peaked the interest of the influential general Jiang Bin and the emperor. Apparently the Zhengde emperor was interested in the world beyond his borders! Unfortunately for the Portuguese, the Zhengde emperor died soon afterwards. Having lost the backing of the emperor, Jiang Bin and the interpreter were executed for corruption and the embassy was given the cold shoulder.
Simão de Andrade was traditionally blamed for causing the embassy to fail by building forts on Chinese territory and kidnapping children for slavery. Some commentators even use this as an early example of Western imperialism in China. Looking at the local situation, however, tells a different story. Since the relaxation of the maritime bans in 1509, southeast Asian traders were grouped by their country of origin and assigned an island for each country. Leaving so many merchants on islands off the coast of China at the time when the Ming navy was in decline meant that these islands were attractive targets by pirates, and Fujitani argues that Andrade's forts must be seen in this light: his forts were not simply imperialistic attempts at annexation, they were meant as protection against pirates in accordance with local practice. What set his forts apart from other traders' was the European artillery. Chinese officials saw the forts as another pirate infestation and became concerned that these pirates with cannons could become entrenched, and so something had to be done. Conservative forces point to this chaotic sea scene as proof that the maritime ban must be reinstated.
Simão de Andrade did not introduce slavery into China. In fact, he only participated in an existing slave trade in China involving children. It was not illegal to buy or sell slaves in China provided that it was government-regulated. However, it was illegal for a Chinese to be sold as a slave to a non-Chinese. Portuguese slave trading activities were therefore done through the black market, and as they were the buyers of Chinese slaves, it is likely that the Portuguese were not themselves the kidnappers of children, but they could have motivated the locals to kidnap children. Also likely was that the increase in demand for slaves accelerated the kidnappings, and in turn gave birth to nasty rumours about Portuguese eating the children.
Surely, if we were just to compare the misjudged letter from King Manuel to Simão de Andrade's reckless actions, then the latter seemed to be the more significant factor. However, the Portuguese should not be singled out for their actions when they were only participating in existing local practices.
Sources: