The short version is that Dreadnought was the culmination of several different design trends driven by advances in technology all coming together into once comprehensive design. These trends included steam turbine propulsion, advances in fire control, and an all-big-gun primary armament. The result was a capital ship that was faster than pre-dreadnoughts (and approaching the speed of armored cruisers), boasting a heavier main armament, and capable of engaging effectively at longer ranges. To top it all off, Dreadnought was a significantly larger ship than the last round of pre-Dreadnoughts, which offered certain economies of scale. For reference, Dreadnought was about 3,000 tons heavier than the preceding Lord Nelson class, 3 knots faster, and it boasted a broadside of eight 12-inch guns as opposed to Lord Nelson's four 12-inch and two 9-inch.

This does raise the question of how we got to this monumental leap, however. Everything up to Dreadnought was an incremental leap, so why the sudden jump? The answer comes in the culmination of incremental improvements in technology hitting an inflection point, particularly when it comes to fire control.

Historically, fire control was done by the gunners themselves - each gun crew would perform their own aiming and correction, and they would largely be left to their own devices with gunnery. With each gun left to fire at its own discretion, gun layouts would rely on slow-firing heavy guns supplemented by faster-firing lighter guns of various calibers. As ships became armored, the heavy guns would provide the firepower necessary to penetrate armor, while the light guns would be able to destroy unarmored or lightly-armored portions of an enemy ship and repel boarders.

This dynamic gained importance with the advent of the torpedo and torpedo boats. With heavy guns having fire rates in the range of a round per every several minutes, there ran the real risk of a nimble torpedo boat dodging a single volley of heavy-caliber fire and closing to engage with torpedoes before a second volley could be fired. This created several different pressures. Smaller, faster-firing guns were needed to engage smaller, faster targets, but the risk of torpedoes also created pressure to engage at further distances.

The issue with engaging at longer distances was that gunnery would start to require some kind of fire control system for any kind of meaningful accuracy. This first manifested in centralized fire control - a single position on the ship directing volleys and providing corrections as the shot fell - but the arms race would continue, seeing increasingly advanced fire control systems designed to push accurate engagement further out. At the same time, spotters watching the fall of shot created pressure to standardize armament, as the myriad calibers of shells each had a different trajectory, and providing fire corrections required the spotters to correctly identify which gun's splash they were seeing.

As these engagement ranges increased, however, the flight time of the shells started to become a meaningful factor. The increased fire rate of a small-caliber gun stops being relevant if the shell is still in the air for several seconds after the gun's reloaded, as aiming corrections can't be applied until the shot has fallen. So while small-caliber weapons remained for anti-torpedo-boat duties, the intermediate range began to fall off as the ranges pushed out. And finally, the same pressures of before - not wanting to mix up which gun's splash you're seeing - comes along to really show the advantage of standardized armament.

So, come the early 1900s, fire control had hit the inflection point where these factors were finally the "right" way to do things. A main armament of nothing but heavy (12-inch) guns supplemented by a number of 3-inch guns to fend off torpedo boats was selected, and an advanced fire control system was included to allow Dreadnought to engage at long ranges. The selection of the turbine powerplant, meanwhile, saved 1,000 tons of weight (compared to traditional reciprocating engines) and left room along the centerline for the "X" turret along the centerline. And to maximize the investment (at the cost of limiting the shipyards that could service it), Dreadnought would follow the trend of the pre-dreadnought arms race by being significantly larger than previous classes of ship.

The result of this was a battleship faster than any contemporary, boasting a heavier broadside, and with a fire control system that made it more effective at range. That meant it could dictate the terms of any engagement with its superior speed, and it was more likely to score the first hits with its superior gunnery.

All this being said, the Royal Navy was hardly alone in recognizing these trends. Battleship evolution was converging on the general concepts we see in Dreadnought, it's just that Dreadnought was first to the party.

While more can always be said, you might be interested in this answer to a similar question.