Many historical sources a full of mythological lands, peoples, and events. At what point do miraculous and fantastic stories render a historical text unreliable?
That depends entirely on what you are using the source for. Primary sources are not just used to get data on what occurred, when, by whom. They also allow us to study and come to conclusions about the society from which those documents spring. Miraculous and fantastical stories provide windows into the world of the author and the reader, provided they are read critically and with care.
Saint's Live, also known as Vitae are a great example of this. Chock full of miracles, visions, etc. as well as cliches and tropes that are often repeated from work to work across a wide range of times, these works serve both as sources of info on the world of the Saint but also (and arguably more so) on the world of the writer and audience.
It can be difficult at times to tell whether the events recounted in a vitae are "true", did Anskar, for instance, actually have the visions which his biographer reports? But regardless of whether these visions were true we can learn a variety of things. For instance, we learn how important the idea of visions were for establishing sanctity and holiness, and of course we get a sense of how medieval people defined "real" or "not-real" and how differently they did so when compared to, say, 20th century people.
Ian Wood's The Missionary Life is a great example of how we do history with "unreliable" documents.