I assume you're interested because you want to make a comparison with gay marriage, so I assume you're thinking about relatively late arguments against it, the ones being made around the time of Loving v. Virginia. If so, the race theorist you want to check out is Carleton S. Coon, who was really the last of the scientific racists, and even early in his career studying race (he didn't start out focused on race), I get the sense he was seen as conservative and out of touch.

So I know that in social science 101 classes, Brown v. Board of Education is often mentioned as the first U.S. Supreme Court case to explicitly cite social science. You should see if you can find not only the original Loving decision, to see what the court cited, but also if possible the brief filed by both sides, to see who they cited (and if they cited any social science).

The world of scientific racism is a bit separate. There are tons of primary documents. A long list would include all the people listed in the "writers" section here. I don't know the secondary literature on scientific racism--/u/restricteddata might be able to recommend a secondary book or two that could orient you and give you an overview (as you see, it's a big field).

As a separate search mechanism, you can find a lot of demographic studies by going to Google Scholar, typing interracial marriage (you'll also want to try other search terms) and then limiting the years, say 1900-1969. I see several studies that jump out at me, "Interracial marriage and divorce in Hawaii", "Interracial marriage in California", "Negro-white marriage in the United States", etc. However, I think these are primarily tracking demographic trends using census data, which is rather limited for measures of well-being, etc. Very little of it seems racist in orientation, however, and I wouldn't expect mainline social science at this time to be promoting segregationist ideas. As far as I know, the arguments against interracial marriage primarily relied on moral, religious, and traditional arguments, rather than social scientific ones based on empirical research. To the degree that they were scientific, they mainly focused on the "naturalness" of racial separation and possibly the difficulty of children to adjust (so you should probably look into the research on "mulattos" as well).

At first glance, the best place to start seems to be the very short review article:

Research on International and Interracial Marriages

Larry D. Barnett

Marriage and Family Living

Vol. 25, No. 1, Family Planning in Modernizing Societies (Feb., 1963), pp. 105-107