He had no evidence, and there's no need to take it remotely seriously. His 1897 book The Authoress of the Odyssey is part speculation and sloppy thinking, part misogynistic garbage.

The core of the argument, such as it is, is (p. 106)

What, let me ask, is the most unerring test of female authorship? Surely a preponderance of female interest, and a fuller knowledge of those things which a woman generally has to deal with, than of those that fall more commonly within the province of man. ...Hence if in any work the women are found to be well and sympathetically drawn, while the men are mechanical and by comparison perfunctorily treated, it is, I imagine, safe to infer that the writer is a woman...

So far this is just sloppy thinking, but it soon transforms into bigotry. On page 107 he invites his read to ask any single women he (sic) knows whether they like having men around the house. Chapter 4 is, I kid you not, about the poet's "jealousy for the honour and dignity of women". At page 136 he notes that "The authoress of the Odyssey is never severe about theft." Chapter 7 is about, and I quote, "Further indications that the writer is a woman -- young -- headstrong -- and unmarried".

Chapters 5 and 6 are devoted to showing that Penelope's character is "whitewashed" in the poem. Butler argues this via a classic case of victim-blaming (126: "One would have thought all she had to do was to bolt the doors as soon as the suitors had left for the night"; 129: "The reader will have noted that on this occasion the suitors seem to have been in the house after nightfall"). He is under the bizarre impression that Penelope's suitors are romantic lovers: p. 128

a man would not have made [the suitors] a band of lovers at all. He... would have made them mere marauders, who overawed Penelope by their threats...

-- apparently without having noticed that they are very much cast as marauders, and having forgotten about their threats of violence throughout the entire epic, their attempt to assassinate Telemachos in a pirate raid, and their actual violence in books 17-18.

It's all supposition without any external corroboration, and based on the presumption that no one is able to depict the opposite sex "without either laughing at them or caricaturing them" (p. 106). No evidence. And no, no one takes it seriously.

The setting of the author at Trapani is more pitiful than anything else. In 1893 he vanity-published a pamphlet containing an article that had previously appeared in a Cambridge magazine called The Eagle in which he laid out the original form of the argument. A postscript shows that he was evidently very annoyed to be told that Drepanon (ancient Trapani) hadn't existed prior to the Hellenistic period (and that is true, by the way). He added a brief counter-argument which consisted mainly of insisting that the archaeologists had better rethink their conclusions.

The argument about the poet of the Odyssey being a woman resurfaced in Andrew Dalby's 2006 book Rediscovering Homer (a book which, remarkably, has a Wikipedia article to itself, in spite of being basically insignificant in research terms; I am sure it has nothing at all to do with the fact that Dalby has been a prolific Wikipedia editor for some years). A 2008 academic review summed up his new form of the argument as follows:

D[alby] has two principal, positive arguments for female authorship. Firstly, the poems show a female sensibility: the Odyssey, for one, "quietly and subversively build[s] a woman's viewpoint into the traditional framework" (152). Secondly, the poems' context argues for female authorship. As D. rightly points out, poems of this scale were not likely to have been recorded from a single performance under normal circumstances. D. thinks a private performance, specially commissioned for the purpose, would have been the occasion for their recording in writing.

The first point is just a re-hash of Butler. If you find it convincing, then I have a bridge I'd like to sell you. The second point may look worth considering, but if it held water, we'd have to apply it also to the Iliad, the Thebaid, the Epigonoi, the Kypria, and several other epics. Basically, we'd have to conclude that every epic longer than 1000 lines or so must have been writen by a woman.

In both Butler's and Dalby's cases, there's no evidence. It's pure speculation based on nothing more than an individual's idea of what "a woman's viewpoint" is (or ought to be) like.

Edit. errors with copy and pasting, added one link.