Scientific racism (as a corruption of Darwinian evolution) definitely played a part, but the outward rationales for European colonisation (British and French anyway) were much more complex than that.

In the British case, the abolition of the slave trade (Slave Trade Act of 1807 [47 Geo 3 Sess 1 c 36]) and later emancipation of enslaved persons (Slavery Abolition Act of 1833 [3 & 4 Will.4 c.73]) led to a kind of funny logic. Before the 1830s, the most prominent abolitionist group in Britain was the Anti Slavery Society. But, when their objectives had been met in Britain and her existing colonies, the ASS was reinvented as the British and Foreign Anti-Slavery Society from 1839, taking the anti-slavery mission global.

Britain - outwardly at least - became the anti-slavery empire. By British colonial logic, the moral capital accrued by ending the trade in human flesh outweighed the moral cost of imposing British sovereignty on large, non-white swathes of the globe. Empire was in many ways considered to be a moral obligation.

The French example is very similar. They were a bit behind Britain in terms of their imperial project, and you couldn't very well have two anti-slavery European empires, could you? Britain had the monopoly on abolition, but France constructed her imperial mission along similar lines. Considering herself the privileged avatar of Enlightenment and republican principles, the French Third Republic (1870-1940) undertook more rapid and more aggressive colonial expansion than under any monarch. The cognitive dissonance might seem incredible now, but then it made perfect sense to spread liberté, égalité et fraternité* throughout the world under the sign of the French gun. * Epic Irony Win : the original 1793 slogan read 'liberté, égalité, fraternité ou la mort'. At least they got the 'death' part right.

Rationales for colonisation changed for both empires throughout the 19th century and material / strategic benefit was a huge part of the calculus, but I've tried to address OP's specific question regarding how European colonial powers justified to themselves inherently illiberal imperial projects with liberal logics in the late 19thC. Early 19thC logics varied -there's a whole other twisted aspect to it where France especially considered the exercise of imperial power necessary for the health of her 'national character' (Alexis de Tocqueville vocally advocated this position throughout the 1830s and 40s).

Two great books to consult are

Isabelle and Robert Tombs, That Sweet Enemy : the French and the British from the Sun King to the present. New York : Knopf, 2007.

and

Jennifer Pitts, A Turn to Empire : the rise of imperial liberalism in Britain and France. Princeton : Princeton University Press, 2005.

During the reconquista in Iberia and the crusades the largest justification for taking this territory was for Christianity. The same thought process could be seen with European colonies in the Americas and Asia where missionaries were set up to convert the native population. That's the reason why Mexico, Brazil, and the Philippines hold most of the world's Catholics. What we see here is an overbearing ideology that European or "western" culture is better than any other. This gave way to racist schools of thought that explained that Europeans had a responsibility to civilize the rest of the world. The U.S. used this same thought process of manifest destiny for most of the 1800s while taking land from the Hispanics and Natives that occupied most of the west. Even Charles Darwin wrote himself that the white man was more evolved than those of other races. This served as the catalyst for most of the colonial period plus access to new markets and resources was pretty good too.