In the UK, to be Prime Minister you first have to be the Member of Parliament for a particular constituency.

On defeat in a general election, or retirement/resignation, most British Prime Ministers have simply gone back to being a local MP, colloquially a "Backbencher", and have often been made a Lord/Lady as a thank you for their services. (Though this has the side-effect of making them less politically relevant, so it is sometimes referred to as being "Kicked upstairs", and several former PMs including John Major have refused a peerage.

Certainly if you look at the career trajectories of the last few ex-Prime Ministers, it seems that none have rejoined the cabinet in another role.

Gordon Brown went back to being MP for Kirkcaldy and Cowdenbeath, participates in discussions and lectures on Globalisation and the Economic Crisis, and is (unpaid) UN Special Envoy on Global Education.

Tony Blair became Middle East peace envoy (to the delight of satirists everywhere), then a senior advisor to JPMorgan Chase, founded Tony Blair Associates which "allow him to provide, in partnership with others, strategic advice on a commercial and pro bono [free] basis, on political and economic trends and governmental reform".

John Major became a backbench MP, president of Surrey County Cricket Club (he was a big cricket fan), and a prolific after-dinner speaker.

Margaret Thatcher spent two years as backbench MP for Finchley, but retired saying she wanted more freedom to speak her mind, at which point she was given a peerage as Baroness Thatcher, entitling her to sit in the Lords, and she continued to do so, while being very vocal in British political affairs despite holding no office, by which time many even in the Conservative Party were worried she was simply becoming a loud irrelevance and damaging her reputation. Eventually she was persuaded in 2002 to take it easy for the sake of her health.

If speculation weren't frowned upon, I would speculate that to others a former PM/President would be seen as a "spent force", "no longer relevant" and so on, while to the former PM/President they may well view any other appointment as a come-down (like a former General being asked to command a regiment as a Captain), or just be tired of the political machine. It's certainly been noted of both Blair and Obama the extent to which they visibly aged during their time in office.

There is also pretty good money to be made as an after-dinner speaker, consultant, and so on. By the time a politician makes it to the end of a term in the highest office, they're usually reasonably old and politics actually isn't that well-remunerated in comparison to leaders of industry, so they may wish to put a bit of money away for retirement/children.