Sakoku tends to be somewhat exaggerated, but the reason that Dejima was handed over to the Dutch from the Portuguese, who you are correct in assuming had a much longer history of involvement in Japan, was because the core cadre of Portuguese people in Japan had been Jesuits.
The transition of Tokugawa ambition from attaining power to maintaining it led to a number of stability-focused policies, including both what we now call sakoku and the outlawing of Christianity. This immediately put the new bakufu at odds with the Jesuits, and after the Shimabara Rebellion was put down (with the aid of guns and gunpowder from the Protestant Dutch at Hirado) the shogunate had finally had enough and expelled the Portuguese from Dejima, moving the Dutch there from Hirado shortly after.
You're absolutely right though that the Portuguese had a huge influence on Japan that continues to this day; tempura and kasutera are just two of the more delicious products we enjoy today from that meeting of cultures four centuries ago.
Source: I'm away from my main library at the moment, but this book by Turnbull (not my favorite in the historiography but serviceable) covers the main thrust of events before and after Shimabara
Ultimately the short and boring answer is that the Shogunate didn't like the religion the Portuguese brought with them, particularly how it challenged traditional culture and loyalties in Japan. The longer answer though makes it clear why the Shogunate was so disturbed by Christianity and more importantly what the Dutch did to alleviate those fears and how the Portuguese completely failed to address them.
The three important bits of context I want to focus on are as follows:
The Tokugawa had only just come to power as Christianity began making inroads into Japan. Naturally they were keen to cement their power and in the early years following the Battle of Sekigahara the Tokugawa reassigned enormous tracts of land to their supporters in order to create a loyal noble class. However the Satsuma, Choshu and many other western domains managed to retain their power in some form and represented a question mark when it came to loyalty to the new Shogunate. In light of this the Shogunate was especially suspicious of developments in the area since if there were going to be any opposition to the Shogunate, it would come from the west.
The Tokugawa came to power at the tail end of the Ikkō-ikki Buddhist movement which had preached resistance against daimyō and traditional structures of authority. Rebel bands caused great amounts of destruction however Oda Nobunaga led a vicious, though successful campaign against them. Christianity faced immediate suspicion from the outset as the Shogunate had just put down one new religious movement preaching against authority and then Christianity arrived stating that followers had to place loyalty to a foreign bishop over that of the rulers of Japan. This all meant that Christianity would not have an easy time proving itself to be unsubversive to Tokugawa rule.
One of the interesting developments in the 16th century was the expansion of Japanese traders over East Asia. In particular those merchants in the Philippines fed back to the Tokugawa how the Spanish were attempting to subjugate the islands with spreading Christianity as their casus belli. This led to concern in Japan that the Portuguese would attempt to do the same which was backed up by English and Dutch merchants telling them the same as well. This was not helped by the Union between Spain and Portugal as it meant that the Japanese felt the Portuguese were actually Spaniards who were pretending they weren't for some suspicious reason. They were not happy when they learnt that both countries shared the same monarch as they had felt previously that Spain's actions in the Philippines may not reflect Portuguese goals in Japan.
With this context in mind, Portuguese attempts to preach their religion in the west of country were seen at best as a challenge to traditional culture and at worst, as an attempt to prepare Japan for colonisation. General persecution began in 1614 after missionaries were expelled (many stayed illegally as "emissaries") and in 1636 the Portuguese were finally expelled to Macao en masse for spreading a subversive cult that threatened Tokugawa rule. The 1637 Christian rebellion obviously cemented the faith as the enemy and in line with that an attempt by Portugal to reestablish relations in 1640 resulted in a ship's captain and sixty others being executed and thirteen survivors being allowed to return to Macao to tell the tale. In my mind there was no way in which Portugal could both preach Christianity in Japan and retain trading relations. This was admittedly due to factors outside of their control (with the main one being Tokugawa paranoia) however for me it is an open and shut case why the Portuguese were kicked out.
For the Dutch, they managed to retain trading relations by having a strict adherence to a trade-only relationship and by repeatedly stressing their homelands' ongoing war with the Iberians which soothed Japanese fears. The Dutch repeatedly stressed that their form of Christianity was very different to Catholicism and demonstrated this by willingly treading on pictures of Jesus and Mary and even firing shots at Hara Castle during the Christian rebellion in the south. Lastly when docking all Christian materials were locked away in barrels on the ship to hide them from inspectors. Even so, the Dutch were ultimately limited to just Dejima so there remained an air of suspicion.
To sum all of this up, Japan was a volatile country at the time Christianity arrived on its shores and Portuguese attempts to preach the faith placed them under immediate suspicion which ultimately resulted in their expulsion as they were presumed to be acting in bad faith. The Dutch successfully alleviated these fears to the extent that they were allowed to trade with Japan, however obviously not enough as they were limited just to Dejima.
Obviously I've glossed over a lot as I've focused just on the religious angle while there were many other (less important but still there) factors. I've also not discussed the assertion that Japan was completely closed to foreign trade as this was mainly imposed on the west.
All of this was sourced from Marius Jansen's The Making of Modern Japan especially the chapter on Tokugawa international relations.
Thanks to /u/KDY_ISD, /u/Bluntforce9001 and /u/ParallelPain for your answers!