As far as i know there's no equivalent in earlier history, so why did Victorian and 20th century nations feel the need to create easily debunked racist theories like phrenology or Nazi eugenics? I get the impression in past eras if you conquered someone and enriched your minority government you simply did so without excuses.
Because it brought legitimacy. Which leads to a new set of questions: 1) why did science suddenly start bringing legitimacy? 2) why did they feel they needed legitimacy of the sort that science brought?
Both of these are actually very interesting and thorny questions about what we mean by the "modern" period, where science emerged as a means of potent political authority and where certain other forms of discourse declined in their ability to establish political legitimacy. None of this was quick in coming, but by the 19th century in the West you have a world where someone with a laboratory can suddenly make the world move on the basis on things they see at the end of a microscope ("anthrax is a bacterium," in the case of Pasteur). You also have a world where scientific claims for running states surpass, say, religious or, to an extent, philosophical (i.e. "we hold these truths to be self-evident...") claims. "Might makes right" might be a subtext, but it stops being the "text" — it loses its ability to be rhetorically/morally compelling in and of itself.
All of which is to say: I think you're coming at this from the wrong end. Scientific racism isn't an anomaly. It's just the modern trend of "scientification" applied to the areas of racism. The same trend was applied to politics, economics, public health, labor practices, medicine, etc. at the same time; racism was just one of the areas that got the "treatment."
This isn't about scientific racism, but if you want a good study on what kinds of social and political institutions get mobilized by scientific authority claims in the 19th century and beyond, Bruno Latour's The Pasteurization of France (from which I get my anthrax example above) is a very excellent sociological-historical study of how these kinds of claims get produced, distributed, and effect the world. On scientific racism, there are many good works but I am partial to Diane Paul's, The Control of Human Heredity (a short, comprehensive, and intelligent overview), as well as Robert Proctor's, Racial Hygiene (about the German situation in particular, and the role of doctors in the Nazi state).