Actually Tsushima is about 50km away from both Korea and Japan. It isn't significantly closer to Korea than it is to Japan.

Territory isn't calculated by how close the island is to one country or another.

Tsushima has historically been held by Japanese military and settlers since the 7th century.

In the modern period, Tsushima has, to this day, been culturally and ethnically Japanese, with residents identifying as Japanese and the Japanese government administrating the island.

/u/AsiaExpert has really answered everything, but I'll reiterate.

First, as previously said, Korea is the mainland region by far the closest to Jeju, whereas the Tsushima islands are right in between Korea and Japan. Your initial assumption isn't really correct.

Ancient Jeju was known to the Chinese as 耽羅 (Danluo in modern Mandarin, Tamna in modern Korean, Tanra in modern Japanese) or by various similar names. The character 耽 (Dan/Tam/Tan) is cognate to modern Korean "seom" island. 羅 (Luo/Na/Ra) is actually cognate to modern Korean "nara" country, so in the end the Chinese name for ancient Jeju was just their pronunciation of Old Korean for "island country". Not unsurprisingly, Chinese records mention plenty of contact between it and mainland Koreans.

By contrast, Tsushima - referred to as 對海國 (Duihai/Daehae/Tsuikai; the last character just means "country" in Chinese) in the Records of the Three Kingdoms - are classified and explicitly stated to be part of Wa or Japan. So already by the third century AD, Tsushima was associated with Japan and Jeju with Korea.

Eventually Tamna was absorbed into Korea and Tsuikai was absorbed into Japan.

This doesn't really change in the following centuries, although Korean kings occasionally attacked it in the 14th and 15th centuries to reduce piracy and there certainly was a Korean influence in the island(s) that has left some relics on the dialect itself. For virtually all of recorded history Tsushima has been far more Japanese than it has ever been Korean.