In short, no, there's no evidence the two met. Vagabond's direct inspiration is Eiji Yoshikawa's 1935 novel Musashi in which Takuan Soho plays the same part.

Kenji Tokitsu's Miyamoto Musashi: His Life and Writings dismisses Takuan's relevance with a short paragraph:

In Yoshikawa Eiji's novel, the young Musashi was strongly influenced by the Zen monk Takuan, but this is purely fictional. The monk Takuan did exist, and he wrote an important text on Zen and the art of the sword, but it is not certain that Musashi ever met him in the course of his life. (pp. 13-14)

Takuan's expertise in the area of Musashi's interest, and the fact they were contemporaries who at times may have been in the same regions, are fertile grounds for creating a fictional connection between the two. It's possible, it makes for a good story, and that's enough for a novel.

The fictional story has caught on so well you'll find plenty of non-academic books talking about how Musashi was inspired by Takuan, or even more boldly, how Takuan was inspired by Musashi. None of this is taken seriously by scholars, and I can't find anyone suggesting the tradition of their relationship goes back further than Yoshikawa's novel.