How did the Netherlands come to and maintain their exceptional relations with Japan during the Japanese period of isolation in the 17th through 19th century?

One thing to keep in mind was that the Dutch were very, very tightly controlled in Deshima, the small manmade island in Nagasaki where they were allowed to maintain a trading post. Dutch (and others who worked for the Dutch East India Company but weren't of Dutch nationality) couldn't leave the island, and Japanese merchants and scholars required special passes and would have been closely watched to enter the compound. In fact, the only time the Dutch staff left the island was every couple years when the head of the post and some of the staff would travel to Edo to pay respects to the Shogun; the Deshima trading post was held to the Alternate Attendance System just as any Daimyo was.

As you also mentioned, the Dutch weren't interested in spreading Christianity. Their main presence in the east was through the aforementioned Dutch East India Company (VOC) which had a profit motive. Much of the success of the Netherlands had come as a result of their excellent skills with regards to trade. By contrast the Spanish and Portugese who had preceded them had permitted a large number of missionaries. In Japan, the primary motive for keeping most Westerners out was two fold, but in a way fell under one umbrella: keeping the system in check under the Shogun. By preventing foreign affairs, Daimyo couldn't gain western allies who could help them foment unrest. By preventing the spread of Christianity, the Shogun kept ideological loyalty on home soil.

So for the Dutch, the reason they were permitted to stay was because they essentially played by the rules. They accepted the Bakufu's very tight restrictions.

For the Shogunate's part, the advantage of allowing this one conduit was that it allowed them to keep abreast of foreign happenings and, more importantly, to keep up with technology. The Bakufu was very open to the circulation of Western books so long as they didn't promote Christianity since they offered advances in things like fortification and gunnery, as well as in things like timekeeping.

Furthermore, the Dutch were one of the few Western powers operating that far East at the time. British influence did not yet spread past India during most of the Edo period, the same goes for the French. The Americans didn't become a player in the Pacific until the 1850s. The Portugese and Spanish did have interests that far as well, but as mentioned before Catholicism tended to come with Spanish interests, and in general the Spanish seem to have been less organized in terms of corporate structure. The Dutch, though, had extensive interests in what is today Indonesia.