To rectify the title a bit, I am also aware of the myriad other puddings the British eat, a heap of which do not come under the sweet dessert umbrella (yum). So, maybe a better thing to ask about is, how come that for example in the U.S or in here, which happens to be Czech Republic, does pudding only refer to the thick and soft, slightly wobbly thingy (I'm not a professional dictionary writer)?
According to the OED, the word pudding is first attested in the 13th century and was borrowed from the Old French boudin, sausage. It originally referred to steamed or boiled entrails stuffed with minced meat and grain. This sense is retained in British English in things like black pudding (a blood sausage) and haggis, which are called savoury puddings.
By the 16th century, pudding was also being used, by analogy, to refer to sweet dishes that were prepared by boiling or steaming a milk and flour-based mixture in a bag, like traditionally made Christmas pudding or clootie pudding. From there it was generalised to mean any boiled or steamed dessert – classic British puddings like jam roly-poly and spotted dick. By the 18th century any consistent definition of the word seems to have collapsed, so you get all sorts of sweet and savoury dishes being called puddings: Yorkshire pudding, pease pudding, bread and butter pudding, rice pudding, and so on. In this surprisingly insightful article about The Great British Bake Off, a linguist is quoted as describing this as an example of a "radial category":
The pudding types all have some things in common with some other pudding types, but not everything, and it’s mostly a matter of convention that they continue to be called 'pudding'.
American English inherited this varied use of the word pudding. Apparently popular British savoury puddings like Yorkshire pudding or black pudding never caught on there and so it makes sense that the word became more exclusively associated with desserts. I'm not sure how it got from that to referring mainly to creamy sweet dishes that would be called custards or mousses in British English.
But the divergence you noted doesn't come from the changing use of pudding in American English, it comes from British English adopting the word to mean the sweet course of a meal. This is a relatively recent thing; the OED has it first attested in the 1930s. Presumably it originated from the simple fact that most of the time what would be served in that course would be a pudding, and according to the article above some people do still make a distinction between a pudding course with a "substantial" sweet dish, and a dessert course with something like ice cream.
As with many recent innovations in British English, pudding-as-a-course started as a way for the upper classes to distinguish their speech from the "lower orders", who would have called it a dessert or a sweet. It then spread to the aspirational middle classes, first in the southeast and then radiating outwards. But it's worth noting that even today it's not something that's found in all dialects. Where I grew up, a pudding (unqualified) would be a Yorkshire pudding, and calling the sweet part of your school dinner anything other than a dessert would get you a hiding at playtime.
You already received a full explanation, but let me add that here in the Netherlands pudding only refers to a dessert as well. It probably carries over to a few more countries here through loan words.