The problem was that there was a distinct lack of trust for the Catholics after the period of missionaries beginning with St. Francis Xavier's arrival on the island. Spanish, but most especially Portuguese traders were arriving in high volume to trade with the Japanese, and at this time Japan is finishing off its period known as Sengoku, when central authority collapsed and daimyos (very roughly analogous to nobles) took control of their own parcels of land. Some semblance of central authority was being restored under Oda Nobunaga and Hideyoshi and the growth of Catholicism was a threat to national unity. Apart from being destructive to the social fabric, many daimyos were picking up the religion too, which could lead to political-religious conflicts in the future. While Oda was a supporter of the Church, Hideyoshi began slowly turning against it until we reach Tokugawa Ieyasu, the final unifier whose family would rule Japan until the 1800s. He undertakes harsh repressive campaigns against the Kirishitans and bans trade with European countries.

The Dutch were able to make it through that prohibition for two primary reasons. First, they themselves were no friends of the Portuguese. They were rivals who would be involved in many wars that the Dutch would eventually come out on top of. Second, their religion was not as strongly concerned with going out and proselytizing the world as the Catholics were in this period. They also already had a strong presence in trading in Asia, something the English didn't really have until later.

The Dutch were trusted by the Shogunate enough to trade on a very limited basis, and based on how well it worked out for them they did not want to mess with a well-established formula until they were forced to.

However, according to the wikipedia article on Nanban trade a few decades after the isolationism was stablished a british ship attempted to reach Japan for trade and was expelled

Small note: This did happen in 1673, but it's not in the Nanban wikipedia article. Perhaps you meant to link another article?

---

The Dutch position as sole Western trading partner, and the lock-out of the British East India company, are in large part a matter of Dutch-British competition and really bad timing for the British.

British trade with Japan began in 1613 when John Saris of the East India Company arrived in Japan. With the aid of his fellow countryman, William Adams, who was working for the Dutch East India Commpany, Saris obtained a trading charter from shogun Tokugawa Ieyasu. The charter allowed the British to trade at any port in Japan. Ignoring Adams' advice to set up a trading post in the shogun's city of Edo, Saris set up a post at Hirado, where the Dutch also had a post.

From the beginning, the British concession wasn't very successful. The Dutch had already saturated the market with Western trade goods, and were not friendly neighbours to the new British traders. In fact, the Japanese authorities had to physically protect the British trading post from the Dutch. Only three years later, the shogunate revoked the British rights to trade at any Japanese port, confining them to Hirado. This was part of a growing trend to carefully supervise trade and contact with other nations. If foreign traders were confined to one port, the shogunate could restrain individual domains from importing weapons or having unsupervised contacts with foreign missionaries/soldiers etc.

The Dutch East India Company went along with the various restrictions and demands of the shogunate, leading to their centuries-long trading arrangement via the small island of Dejima in Nagasaki. But the British weren't making a profit. The Dutch weren't just competing with them in Hirado itself; on the seas, Dutch ships regularly attacked and captured British ships on their way in and out of Japan. In 1623, the British had had enough, and announced they were closing their post at Hirado, but might return to Japan in the future.

As a result, the British did not have any foothold in Japan when the Portuguese were expelled in 1638. They were completely absent, unable to defend their interests, as the restrictions on trade and travel hardened. There were many British discussions of possibly returning to Japan over the following decades, but nothing came of them until 1673. At that point, the shogunate's arrangement with the Dutch had been solidified, and the British were an unknown quantity. The Dutch were not happy to see them arrive either.

William G. Beasley writes of this British attempt to re-establish trade:

The seclusion policy rapidly became a tradition, and by the time Simon Delboe entered Nagasaki in the Return in June 1673, the Japanese needed little persuasion to refuse all requests for trade. The Dutch, who had received early warning of the English plans, were careful to point out that by Charles II's Portuguese marriage England had entered into alliance with Japan's traditional enemy. This provided an admirable reason for refusing to honour the privileges Ieyasu had granted in 1613, a copy of which Delboe brought with him. After weeks of waiting at Nagasaki, Delboe was informed that he could not be admitted to trade, and Japanese officials held out little hope that success would attend any future English efforts. Once given, they said, the Shogun's decision was irrevocable'.

• xvii-xviii, Great Britain and the Opening of Japan 1834-1858, William G. Beasley, 2013 Edition

Only fifteen years later, after the "Glorious Revolution", Britain had a Dutch protestant king, William of Orange. Beasley points out that this seems like it would have been the ideal time to re-establish British trade with Japan. But despite some discussions of the idea, no one made any serious plans. The British East India Company had settled into the status quo, and trade with Japan had never made much money historically. So, the British let the issue drop, and it wasn't till 1791 that they attempted again to open up trade.

From the late 18th century onward, Western powers increasingly attempted to re-open commerce with Japan. Over the past century, the Dutch arrangement had suited Japan well, so there wasn't any rush for Japan to openup to the unknown. The colonial ambitions of these Western powers had also become a matter of concern for Japan. Of all the Western powers, the shogunate most worried about Russia, which was beginning to encroach on the northern islands claimed by Japan, so their cool treatment of Russian embassies is very understandable.