One answer from economic history: prices.

Ewout Frankema, Jeff Williamson and Pieter Woltjer have a 2018 paper in the Journal of Economic History suggesting that part of the motive for the scramble for Africa was a run-up in relative prices for the commodities that could be produced there, combined with a decrease in the costs of colonization. Taken together, the economic incentive to take direct control over African countries increased during the period, redirecting the attention of colonial powers towards the region.

To back up a step: From the late 18th century until around 1840, Europe went through the beginnings of the Industrial Revolution. Mechanised production had diffused thoroughly enough to be driving down the cost of manufactured goods, making them seriously competitive in export markets vs. locally produced goods. This is the approximate inflection point that (among others) Steve Broadberry and Bishnu Gupta find for British textile exports to India, and Manuel Llorca-Jana finds for the southern cone of Latin America. So, for a few decades, you have broadly emerging industrialised production in manufactured goods, and increasingly broad industrialised transport infrastructure (rails, steam shipping, telegraphs, etc).

The industrial revolution is doing its job a little *too* well, and this is driving up both the price of inputs into industrial processes. Rising incomes led to demand for consumption goods, pushing up their prices in the North Atlantic. This means the volume of industrial goods flowing out of Europe was very high, but their relative price was driven lower and lower; the terms of trade were changing in favour of primary commodities. This increase begins in the 1840s, peaks around 1880, and then falls steadily in the 20th century. (Incidentally, it is the second half of that chronology that inspired the famous Prebisch-Singer hypothesis that the terms of trade fall over time against primary producers. But they were measuring from the early 20th century; had they been measuring from 1800-1900, the conclusion would have been precisely the opposite.)

These high relative prices are an incentive to increase production of these products. You can see, across the 2nd half of the 19th century, a worldwide push towards increasing the volume and scale of primary exportables. Assam gets its tea plantations, Sao Paulo its coffee, Manaus its rubber, Russia/Canada/US/Argentina its wheat, Australia its wool, and so on. Africa in particular was a (in some cases figurative) gold mine for primary commodities. Colonial projects which in the 18th or early 19th century would have seemed prohibitively expensive and not very remunerative to investors suddenly seem at least potentially profitable. And so, there is a scramble - the economic costs of colonialism are down, and the economic rewards are up, so the European powers (approximately simultaneously) abandon their previous trade-centric approach and instead move towards direct colonisation.

One can also tell stories about the rise of industrial states with military capacity (guns, artillery, gunboats, etc), about the development of medicine (especially quinine against malaria), and about the rise of "free" trade and the decline of tariffs. But I know less about those stories.