This is a great question! In a nutshell, Robert Walpole managed to gain power and retain it for so long because he was an innovative politician, willing to transform British political norms in order to stay in charge. But in my view, he was never quite as secure in his power as historians have traditionally argued.

First, let's talk about how Robert Walpole gained power in the first place: he was strategic, but he also got really, really lucky. Following the Hanoverian Succession in 1714, the Whig Party (of which Walpole was a member) began to fracture in two, an event that is known as the Whig Schism (c. 1716-1720). On one side of this schism were the Whig ministers James Stanhope and the Earl of Sunderland, who were supported by George I: they wanted to use Britain's military as a tool to expand British commerce to new European and imperial markets, and they therefore involved Britain in the Great Northern War (against Sweden) and the War of Quadruple Alliance (against Spain.) On the other side of the schism were Charles Townshend and our buddy Robert Walpole, who thought that this militant foreign policy was a waste of British "blood and treasure," and that a commercial nation's best policy was to always maintain the peace. For a period of time, Walpole was actually kicked out of the Whig Ministry for his foreign policy positions. In 1720, the two sides of the party reconciled and Walpole was allowed back into the Ministry, but relations remained uneasy between the two sides.

Now for the lucky part: the South Sea Bubble, when a state-sponsored company, the South Sea Company went belly up, putting the British state's finances in a precarious position. While this might not seem to be a "lucky" occurrence, for Walpole it almost certainly helped rather than hindered his political career. For one, he had been kicked out of the Ministry at the time of the bubble's peak, so most Britons blamed his opponents Stanhope and Sunderland rather than him for the financial disaster. (There were exceptions: Cato's Letters, a series of insanely popular newspaper articles, put the blame at Walpole's feet.) For another, Walpole had a lot of financial acumen, and through the creation of some new financial apparatuses, including a Sinking Fund intended to lower the national debt, Walpole was given credit for putting the nation's finances back on solid ground.

There was also a second series of lucky occurrences for Walpole: the death of almost all of Walpole's serious Whig political rivals. From 1721-1722, James Stanhope, the Earl of Sunderland, James Craggs the Elder, and James Craggs the Younger all died in quick succession. (For the curious, they died of stroke, unnamed illness, smallpox, and suicide, respectively.) So Walpole was almost the only Whig politician left standing. As you might imagine, this put him in a good position for gaining power over his party, and therefore over the British state as a whole.

Still, it's one thing to be an important Whig minister, and it's quite another to basically invent the position of Prime Minister. So here we have to give Walpole credit: he was a great politician. There were several policies he pursued that allowed him to consolidate so much power.

1. He perfected party discipline: In previous administrations, there wasn't really a mechanism for punishing party members who voted against the administration. Robert Walpole was one of the first politicians to recognize that the Ministry's control over government patronage gave the ruling party a lot of power to force their members in line. By using a system of carrots and sticks (rewarding and removing offices and sinecures), Walpole made sure his fellow Whigs toed the party line. At election time, he also distributed slates of candidates approved by the government to ensure that voters in certain districts voted as a block. And he regularly checked in with all of his party members through weekly meetings when Parliament was in session at a meeting place called the Cockpit. These tools may seem like basic methods of party politics to us now, but they were extremely innovative in the eighteenth century.

2. He expanded the Whig Party: Robert Walpole was strategic about trying to incorporate the Tory Party's base of landed, rural gentlemen into the Whig Party. For instance, he tried to shift the tax burden away from land towards commodities. This sometimes got him in trouble, as when England's commercial cities such as London revolted against his attempted excise tax on wine and tobacco in 1733. But overall, Walpole managed to add a lot of former Tories to his party infrastructure.

3. He prioritized England: This is a bit more controversial, but I would argue that Walpole recognized that he could better retain power if he prioritized the interests of Englishmen over the interests of Britons who lived in Scotland, Ireland, and the American colonies. After all, Englishmen could vote in parliamentary elections, while American and Irish subjects could not. (Scots could vote, but they were underrepresented in the Westminster Parliament.) Walpole often took a heavy hand in governing these outer regions of the British Empire. To take the example that I know best, in Scotland, Walpole cracked down frequently on smuggling and tax enforcement, even though Scotland's economy was far less developed than England's during the 1720s-30s. Walpole even sent the British army in to occupy Scotland on two occasions: the Malt Tax Riots of 1725 and the Porteous Riot of 1736. He would have been loathe to take such a militant approach in England.

As I mentioned at the beginning, Walpole's power was never as secure as some later historians would believe. The Tory Party and the new Patriot Whig opposition (who had splintered off from the Whigs because of their opposition to Walpole) were fervently opposed to the direction Walpole was taking the country, and they launched a formidable resistance against his rule. They published popular newspapers, pamphlets, and plays that ridiculed Walpole as a tyrant, a thief, and an overweight consumer of the nation's spoils. It helped that almost all the best writers of this period--Jonathan Swift, John Gay, Alexander Pope, etc.--had extremely anti-Walpole politics. Because of this political resistance, Walpole came extremely close to losing his majority in Parliament in 1733 when he failed to pass an unpopular excise tax. A few years later in 1739, Walpole was forced into a war with Spain because of popular agitation against his pacific approach to foreign policy.

Finally in 1742, after several years of mismanaging a war that he never wanted to fight in the first place, Walpole was finally forced to step down from the position that he had essentially invented, Prime Minister. But the fact that Walpole managed to maintain power for two decades certainly speaks to his political shrewdness.

In sum, Walpole gained power because he was lucky, but maintained it because he was a great politician.

Sources:

J. H. Plumb, The Growth of Political Stability in England, 1675-1725 (1967)

J. H. Plumb, Sir Robert Walpole (1956)

W. A. Speck, "The Whig Schism under George I," Huntington Library Quarterly (1977)

Julian Hoppit, "Scotland and the Taxing Union," Scottish Historical Review (2019)

Linda Colley, In Defiance of Oligarchy: The Tory Party (1982)

Steve Pincus, The Heart of the Declaration (2016)

Paul Langford, The Excise Crisis: Society and Politics in the Age of Walpole (1975)