Further, how did the segregation and anti-miscegenation states' courts argue this in context of the "separate but equal" doctrine/ideology?
While this isn't perhaps the idea you had in mind, I'm going to focus on one specific aspect of anti-miscegenation laws that were probably most disproportionately applied with gender in mind, namely white men who had married women that legally speaking were not "white" as defined by the race laws of the time, or even might be construed as not doing so, given how such laws of the period were written. This wasn't a situation of the state pursuing the couple though, but rather the husband using this information to seek the dissolution of his marriage in a time when divorce was rare and allowed in only a few narrow circumstances.
An example of this can be found in Kirby v. Kirby, a 1921 case in Arizona, a state which at that time banned interracial marriages. Joe Kirby had been married to Mayellen Kirby for seven years at which point the marriage had broken down. Instead of trying to pursue a divorce, he decided to instead try and have the entire marriage invalidated by claiming that his wife, in fact, was legally black. It is interesting that the defense never tried to prove the contrary - that she was legally caucasian, and instead tried to call into question Joe's parentage, which was Spanish/Mexican, but which was rejected by the judge since:
Mexicans are classed as of the Caucasian Race. They are descendants, supposed to be, at least of the Spanish conquerors of that country, and unless it can be shown that they are mixed up with some other races, why the presumption is that they are descendants of the Caucasian race.
Mayellen was given no such benefit of the doubt, her "guilt" merely granted for her "dusky countenance" which gave off the "distinguishing characteristics of the African race and blood". Although appealed to the Arizona Supreme Court on the grounds that the law was unconstitutional, they refused to see it that way and upheld the annulment of the marriage on racial grounds.
Now, the question you of course might be wondering is "did Joe Kirby know in advance?" but that is almost beside the point, as it was never proven that Mayellen had anything more than some features which made her look black, which the court simply took to be enough. Perhaps she was, perhaps she wasn't. Perhaps Joe knew from the start, perhaps he didn't. What is considerably more important is how he was able to use the possibility of it as leverage to get out of a marriage he no longer wanted to the clear and severe detriment of his former wife.
Another such case decided under the law was not for a divorce, per se, but similarly used as a means to deprive a wife of her share of the marital property. Marie Antoinette Monk had been named in a will by her husband Allan when he died, but an earlier will had named a friend Ida Lee, who took to court to try and prove that Marie was part black and thus the marriage - and the will - were legally invalid. Based on pseudoscientific testimony, it was determined she did in fact have 1/8 "negro" ancestry, and thus was in violation of the law, and had perpetrated a fraud on her husband by concealing it. She too attempted to appeal to the Arizona Supreme Court, similarly to Mayellen arguing that the law as written essentially prevented a person of mixed race from marrying anyone, but the court refused to entertain the argument (which is relevant to your secondary question. In short, they avoided answering it).
Even in states which lacked anti-miscegenation laws this was not unknown in the courts. A 1924 case in New York saw Leonard Kip Rhinelander seeking an annulment of his recent marriage to Alice Beatrice Jones on racial grounds, and although he couldn't appeal to a law making such a marriage on its own illegal, he argued that she had concealed her heritage from him. The resulting trial was quite salacious, involving Alice partially disrobing for the jury to prove the defense's charge that Leonard had to have known in advance. In the end it was part of a successful strategy, the jury deciding that she had not deceived him. The jury had seen through Leonard's ploy to dissolve the quickly regretted marriage, but not before Alice has been forced to go through a quite shameful process to refute it.
Even in states where such laws did exist though, it wasn't always a given that a husband's attempt to skip out of a marriage in this way was successful. North Carolina, which had a similar law, saw the case of Ferrall v. Ferrall in 1907 begin for similar reasons to Kirby, but end quite differently. Claiming to have been ignorant of the fact his wife Susie was "of negro descent within the third generation", Frank Ferrall used this claim to seek an end to the marriage. Susie claimed that she had no black ancestry, but did concede she might have indigenous, or Portuguese, ancestry which might contribute to her slightly dark complexion. The focus of the trial came to be her great-grandfather, the "racial taint" her husband alleged, but to satisfy the law which only looked to the 3rd generation, it needed to be proven that he was ""a real negro [...] that did not have any white blood in him." The jury did not find this to be proven, but more importantly, chastising Frank from the bench for the route that he chose to try and pursue, when the case went to the Supreme Court, Chief Justice Walter Clark wrote:
The [husband] by earnest solicitation persuaded [Mrs. Ferrall] to become his wife in the days of her youth and beauty. She has borne his children. Now that youth has fled and household drudgery and child-bearing have taken the sparkle from her eyes and deprived her form of its symmetry, he seeks to get rid of her, not only without fault alleged against her, but in a method that will not only deprive her of any support while he lives by alimony, or by dower after his death, but which would consign her to the association of the colored race which he so affects to despise. The law may not permit him thus to bastardize his own innocent children [...] but he would brand them for all time [...] as negroes—a fate which their white skin will make doubly humiliating to them.
Although Clark didn't take on the law itself directly, he even went to far as to hint that Frank's choice was an immoral one even if he had been proven correct:
If indeed the plaintiff had discovered any minute strain of colored origin after the youth of his wife had been worn away for his pleasure and in his service, justice and generosity dictated that he keep to himself that of which the public was unaware.
In most of these cases, there is the consistent strain of the law being used as a tool by men to escape responsibilities to the woman in their life. Even in cases where a divorce might have been possible via a different avenue, in seeking to invalidate the entire marriage and label the woman as a fraudster, it removed any further legal or financial support that the man might otherwise have owed. Pascoe, in her piece which focuses on Kirby v. Kirby, ably sums up the circumstance and aim of his suit:
As long as miscegenation laws regulated marriage more than sex, it proved easy for white men involved with women of color to avoid the social and economic responsibilities they would have carried in legally sanctioned marriages with white women. By granting Joe Kirby an annulment, rather than a divorce, the judge not only denied the validity of the marriage while it had lasted but also in effect excused Joe Kirby from his obligation to provide economic support to a divorced wife.
Even in the case of Monk, where the suit was brought by another woman, it is unavoidable to see similar themes in play, the case peddling in racialist, racist, and heavily gendered thought about black jezebels. It also was a tool for retaliation. Ernestine Baker, for example, had every reason to want the divorce she sought, on the basis of his drunkenness and 'extreme cruelty' that so quickly became apparent mere weeks into their marriage. The court agreed with her, and initially granted her alimony, as well as the car. Not wanting to part with any of that, Billy Baker alleged that the marriage was invalid since she had black ancestry and that he was a "full blood Seminole", which violated Oklahoma law. The Oklahoma Supreme Court agreed with him, remanding it back to the lower court with the order to weigh the racial issue, which would invalidate any alimony settlement.
There was a flipside though. Rosina and Cezar Calma had married in New Jersey, legally, and moved to Virginia, where they lived in violation of the law as she was Filipino and he Caucasian. Although they were not prosecuted and left alone - common in Virginia because it didn't involve a black and white marriage, which the law mostly focused on - when they decided they wanted to divorce, the court refused to grant it as it refused to recognize they were married. When appealed to the Virginia Supreme Court, as with Arizona is sidestepped Constitutional issues, rejecting their appeal on procedural grounds and refusing to weigh their argument that Virginia was in violation of the 'Full Faith and Credit' clause in the Constitution and the 14th Amendment as well.
So in short, while this is a fairly narrow slice of your question - and to be sure there is much more that can be said about how interracial relationships were (and continue, but that is another matter) viewed based on the respective genders - I hope that it does provide an interesting, if tragic, illustration about one particular way in which the law was applied, in in particular, a way in which is was generally used along clearly drawn genderlines.
There are other interesting avenues to pursue here, and please do check back this evening as if I have time later today I may write some more from new angles as well, but for now I'm off to see MIB:I.