The trickiness of this question is that you haven't indicated what theories you are considering "scientific racism," nor how one decides what counts as "fringe." This is not a pedantic point: the people who were scientific racists did not label themselves as such, and deciding who is "mainstream" and who is "fringe" is a difficult activity. So depending on when and who you are talking about, some were definitely fringe — William Shockley's approaches to race and eugenics, were, for his time, pretty fringe. And if one's definition of "scientific racism" is very broad (e.g., labels every scientific belief we woul dconsider racist today as a form of "scientific racism") then it becomes a not-very-useful inquiry (in such a case, Darwin's theories about "savages" would fall under such a category, even though Darwin was, relative to his contemporaries, relatively mild in these areas).

But if you are asking, were there prominent scientists who advocates views that are unambiguously scientific racism — for sure. Many. Louis Agassiz, for example — a major "mainstream" scientist in Darwin's time, who used his "scientific" theories to advocate for explicitly racist ends (justifying slavery, etc.). The entire field of Victorian anthropology, for another, and physical anthropology generally into the early 20th century. Almost all studies of human genetics in the early 20th century were forms of eugenics; not all were racist (some were merely classist), but many were.

This is not to say that there were not also counter-trends among intellectuals. The Victorian anthropologists, for example, were opposed by the ethnologists, who were far more liberal in their views on race. The physical anthropologists were eventually displaced by the Boasians. Agassiz was opposed in his theories by Darwin, and others. But these were still "mainstream" ideas, and at times they definitely dominated.

Moreover, the assumptions about racial superiority/inferiority that underpinned these beliefs were widespread not only in academia, but many other domains of society. These were deliberately exclusionary institutions, dominated by whites, and it was the rare thing to question why that was.

So no, I would not call it a "fringe movement," though there were certainly specific theories or assumptions that were more fringe than others.