I question your premise. In many cases (ships, locomotives) the shift to oil happened significantly later than the turn of the century, and where oil/diesel was widely used earlier (aircraft, automobiles) there were no serious steam equivalents to compete with in the first place. In regards to stationary power generation, thermal power plants are still very much a thing today due to the abundance of coal.
Also, you seem to be confusing steam with coal. Many WWII ships had oil-fired steam turbines; only after WWII were combustion engines widely used in ships.
Still, the short answer is that oil and combustion engines are simply better from an engineering viewpoint in a number of ways (less maintenance, higher energy density, miniaturisation) and coal-fired steam engines had already pretty much reached their full potential by WWII. For a more in-depth answer you should probably first clarify what you're actually asking.
Heat.
The invention of the first really practical steam engine was due to the insight of James Watt . He approached Edinburgh University chemist James Black, curious as to why Newcomen engines seemed to consume so much coal. Black was able to clue him in; Black had discovered latent heat, hidden heat: when water was heated to steam a lot of heat was needed in making the liquid into a gas. Watt saw that a Newcomen engine would alternately heat and cool the steam cylinder, and so a lot of heat was being thrown away. Watt redesigned it, so that the steam cylinder would stay hot, and a separate condenser would stay cold. Heat was thus conserved, steam engines could be smaller, smaller engines could then be put on boats to make steamboats, or on rail to make locomotives. But there was still some confusion as to how efficiently a steam engine could run. That was answered by Sadi Carnot in 1820. Applied to steam engines, the Carnot Cycle showed that , generally speaking, hotter steam and colder condensers were better. But also that the ideal steam engine was adiabatic- that is, it did not lose heat to any of its surroundings.
Knowing this, the 19th c. steam engine designers tried to make a steam engine hotter and tried to keep the heat from being lost. They discarded the earlier "atmospheric" engines, pioneered by Watt, that worked at low temperatures, and went to high pressure , higher temperature engines. . Instead of feeding the boiler cold water from outside, they'd pass the cold water first through the jacket of the condenser, to pre-heat it. Boilers were configured to grab more heat from the hot gas of the fire as it passed through to the chimney. Triple-expansion engines were invented, to try to capture all the expansive force of the steam before it condensed. However, in the end, despite very ingenious and sometimes complex design changes, the reciprocating steam engine really just leaked energy all over, and could not capture the last energy of the low-grade heat in the warm water coming out of the condenser. W.E. Dalby drew an excellent diagram of the heat losses ( it's here: on P. 44) Starting with a stream of energy 100 units wide for the fire, 5.5 % is lost in radiation from the boiler, 11 % goes up and out the chimney, the feed-water heater radiates off 2.7 %, but 6.4% is recaptured ( grabbing a little heat from condensed steam) 2.45% radiates off from the engine. Only 14.9 % of the heat does actual work: the remaining 60% is sent off with the condensing water and is lost.
While the early diesel engines were not as efficient as they are today ( a theoretical 45%) , they could do better than the 14.9% of a reciprocating steam engine. They also proved to be somewhat easier to make, maintain, install, and adapt. There are a lot of gizmos in a steam engine: pumps, valves, gauges, feed water injectors, hot wells, oil cups, oil traps... When the railroads began to transition from steam to diesel in the mid 20th c., the repair shops would say that with a steam locomotive it took 5 minutes to know the problem and 3 days of work to fix it, but with a diesel it took 3 hours to figure out the problem and an hour to fix it. Internal combustion engines also took up comparatively little space for the horsepower they provided.. Small, simple "hit and miss" gas engines were manufactured and sold to countless fishing sailboats around 1900. They provided a welcome and pretty reliable source of power to get the boat back into port in a dead calm, and took up far less space below deck than a steam plant.
But steam did not go away. Not being stupid, engineers like Charles Parson realized that a steam turbine could run much more efficiently- turbines could use steam at quite high temperature/ pressure. They could also permit the steam to expand to the lowest possible pressure- something that was impossible to do in a reciprocating engine because, well, you'd need an impossibly big piston and cylinder. Efficiencies in steam turbine power plants by 1912 were already reaching 68%, and they are still here.
William Ernest Dalby Steam Power
The steam engine is still very much with us. It's been continuously improved over the last century. Developments since the late 19th century include the steam turbine, new fuels (including nuclear), multi-fuel boilers, and general improvements in reliability and efficiency. The best steam engines are very efficient thermodynamically. Efficient enough so that there isn't much room for improvement - their efficiency is close to the theoretical thermodynamic limit. This is an important advantage, since it means you can get more output for less cost in fuel. It is important to note that many old steam engines designs - which can still be found in use - are not very efficient at all. The improvement has been large.
The internal combustion engine has some advantages over the steam engine:
Better power-to-weight ratio. A steam engine needs the actual steam engine proper (the turbine or the piston/cylinder) and its fuel, which is similar in weight to what is needed by an internal combustion engine, and in addition to that, a boiler, the water in the boiler, and either a condenser system to recycle that water or even more water. While a large steam turbine engine can have quite modest-in-size turbines, the boilers can dwarf the turbines in size and weight (if you happen to be in San Diego, you can visit the Midway Museum (i.e., the aircraft carrier) and see this for yourself - the triple-expansion turbines are surprisingly small for the size of the ship). Thus, where compactness and weight matter (e.g., cars, aircraft, lawnmowers, chainsaws), internal combustion engines are typically better than steam engines (and if short range/running time is acceptable, or plug-in electrical power is available, electric motors might be even better).
Responsiveness. With a steam engine, the water in the boiler must be heated to obtain steam at the desired pressure. This takes time, even if the boiler is very small. For a large steam engine, this can take many hours. Steam engines are unsuited to stop-start operation, compared to internal combustion engines and electric motors (taxis and buses are much better candidates for steam engines than the privately-owned commuter car). A commuter car in which you have to start heating your boiler at least 20 minutes before you can drive it will annoy owners.
Of the three types of engines/motors noted above, each has its own advantages and disadvantages:
Steam, advantages: can be efficient, able to use a variety of fuels, can be made to have low emissions, easier to scale to very large size when very high power is needed.
Steam, disadvantages: heavy and large, takes time to reach operating pressure, old designs are inefficient.
Internal combustion engine, advantages: can use a variety of fuels (but a smaller variety than steam engines), especially gas turbine engines, good power-to-weight, excellent energy density and easily-handled fuel (it can be pumped), easy to turn off and on quickly.
Internal combustion engine, disadvantages: can be noisy, can have undesirable emissions (e.g., nitrogen oxides) due to high temperatures.
Electric motor, advantages: compact, quiet, electric power can be generated efficiently by steam engines, very easy on-off operation, easy to scale to very small size.
Electric motor, disadvantages: power supply - batteries are heavy and provide very poor energy density compared to typical fuels used by steam engines and internal combustion engines; connection to the power grid avoids the need for batteries, but limits mobility.
As a result, we see many electric motors within the home (vacuum cleaners, blenders, fans, etc.), and some electric vehicles, especially short-range electric vehicles. Steam engines dominate in large fixed installations where the engine can operate continuously (e.g., electricity generation using fossil fuels or nuclear fuels is dominated by steam engines, and there are electricity-generating steam engines with solar-heated boilers). For vehicles where power-to-weight is important, and electric motors would not provide sufficient range or operating time with a reasonable weight of batteries, the internal combustion engine rules.
There are two interesting vehicles worth further brief discussion:
Railway locomotives. While the old-fashioned steam engine is largely obsolete (it still sees use in coal mines in China, due to being able to use the coal as fuel), electric motors, steam engines, and internal combustion engines (diesel) were all in use for much of the 20th century. Electric motors were favoured for urban networks, where electric power was readily available, and especially for underground lines where the lack of emissions was a very important advantage. Steam engines were favoured for long-distance haulage, due to cheap fuel and efficiency. Diesel engines were favoured for (a) stop-start operation, e.g., in marshalling yards, (b) lower emissions for urban rail, compared to coal-burning steam, and (c) long-distance haulage in desert regions. The last point was due to difficulty in providing water for the boilers in deserts; the alternatives were condenser systems to recycle the water, which were bulky (especially in hot climates), and electrification of long distances of line: diesel won.
Ships. For long-distance travel, the engine can run continuously, avoiding the stop-start problem. When the main competing types of engine were reciprocating (i.e., piston-and-cylinder) steam engines, steam turbines, and diesel engines, steam dominated large ships into the 2nd half of the 20th century. An important part of this was the difficulty of building reliable and affordable very large diesel engines. For small ships, diesel could be used, but for large ships, steam was necessary. If high speeds were needed, steam turbines were often chosen. Ease of handling fuel resulted in a general shift from coal to oil as the fuel, and modern designs of engine use turbines. In the early days of competition between steam and diesel, the fuel (coal, heavy oil) used by steam engines was cheaper, which compensated for lower efficiency (the things you need to do to get very high efficiencies as obtained in steam turbines for electricity generation require large boilers, and work best for very large engines). Improvements in efficiency of diesel engines, and relatively cheaper fuel, and especially improved reliability meant that diesels took over much of the former role of steam engines (and pretty much eliminated the reciprocating steam engine as a serious contender for marine propulsion). This left the steam turbine and diesel engine as competitors. The development of the gas turbine, essentially the internal combustion equivalent of the steam turbine, allowed the boiler to be eliminated, and the gas turbine has been pushing aside the steam turbine in marine use over the last few decades. However, even very early in the time when these types of engines were competing, some types of ships/boats used diesel. Submarines switched to a combination of diesel engines and electric motors very early in their development - the engine must be shut down for submerging, and this is much, much easier with diesel engines than steam engines. Since submarines are often quite small, the compactness of diesel engines is also useful. The electric motors are for use underwater. The steam engine did make a comeback to submarines: the nuclear submarine. Since the nuclear-powered steam engine doesn't need oxygen from the atmosphere for operation, it can be used underwater, and doesn't need to be turned off for submerging. Small boats also benefit from the compactness of the internal combustion engine, and are often operated stop-start.
In summary: steam - in the form of very efficient and very large steam turbines, largely for generation of electrical power - lives on. Mobile steam engines were typically much less efficient than the large stationary turbines used in electricity generation. Despite this, the reciprocating steam engine remained in common use on railways and in ships into the 2nd half of the 20th century, but was outcompeted by diesel, as diesel engines become more reliable. Steam turbines remained common and competitive for some roles in shipping to the end of the 20th century. Steam propulsion continues in nuclear-powered ships.