Appeasement, as a policy, assumes that the party being appeased has a limited set of demands and that meeting those demands will make the appeased party back down from further demands. Simply put, that was a false assumption when it came to the Western powers' interactions with Hitler.

This was the basic idea. Hitler claimed that he was only demanding justice for the German people (both inside and outside of Germany). To this end, the Western powers were willing to appease Hitler so long as Hitler's moves seemed reasonable. When Hitler re-militarized the Rhineland, that was seen as a perfectly reasonable action, despite it being in violation of the Versailles Treaty. The Rhineland is German territory, after all, why shouldn't the German army be allowed to go there? When he occupied Austria, the same thing. Austrians are ethnic Germans, and despite what they'll tell you today, there was a broad base of support (although certainly not the high 90% that was claimed in the plebiscite) for the German occupation of the country. In Munich, Hitler demanded only the Sudetenland, which was populated mostly by ethnic Germans. This was, after all, one of the main articles of Wilson's 14 points - self-determination. Why shouldn't Germans be governed by Germany? Yes, it's deeply unfair to the state of Czechoslovakia, but when you're trying to avoid a World War, you make ugly concessions, which is what happened at Munich.

The reasonableness (and I'm speaking from a purely pragmatic point of view, not from a moral point of view) ends with Hitler's invasion of the rest of Czechoslovakia in early 1939. There really is no defense of that and, if anything, it tipped the Western allies off to the fact that Hitler was not really making reasonable demands anymore, nor would his demands cease, but rather he was engaging on a course of expansionist conquest in East Central and Eastern Europe. They, the Western powers, drew the line at the next step - the invasion of Poland. I can't remember where, off the top of my head - I think it's in his Table Talks - Hitler even mentions to his subordinates that he was afraid that the Western Powers would appease him in the invasion of Poland, thereby not giving him (Hitler) the war that he really wanted.

To answer your question directly, appeasement didn't work because Hitler never had as his end goal the motives which he proclaimed publicly at the time - incorporation of Germans into Germany and self-determination for Germans living outside of Germany. Rather, his goal was an expansionist war in the East against Poland and especially the Soviet Union. Once it became clear to the Western allies that these were his real goals, appeasement as a policy ended and the war began in earnest.

I think that to answer this question we need to look a few years back. Europe has for centuries functioned with many states which accept the legitimacy and sovereignty of each other. Up to WWI, the general consensus was that there needs to be a balance of power in between European states. This was achieved to a large extent after the end of the 30 year war and held up for many years (with the exception of Napoleon, but that would be a topic on its own).

This dynamic balance, which successfully kept Europe without large scale conflicts, was mainly based on the fact that there were many smaller German states. These states alone were too weak to be able to threaten another state, and if needed, they could've easily formed an alliance strong enough to defend themselves.

This was very important for the whole of Europe - but this changed drastically after Germany united into one state. They easily became the most powerful state in Europe, with the slight exception of English naval dominance, however even that was threatened.

The balance has changed - Germany suddenly could've reached much further than before, considering both colonies and their main state borders.

This and a few other factors then resulted in WWI, after which the treaty of Versailles was signed. This is the most important factor, and has been described as one of the worst treaties ever signed in history. The treaty itself should've put balance of powers back in place, however it did not consider the fact that Germany still was a powerhouse. The treaty had no one to overlook and guard it. France was decimated by the war, the British took a step back from Europe, Italy became an ideological ally and both Czechoslovakia and Poland lacked the political and military power to be able to really punish Germany if they did not play along.

Another option was the actual use of force in some form of "preventive strike". With France lacking the power and England so opposed to war, there was no possible coalition that could strike fast enough to prevent any German attempts to gain power. During all this, Germany still had the same capacity as it had before WWI, if not greater. That meant that any possible appeasement wasn't relative to the actual power of Germany (as we saw during WWII, when Germany almost fully controlled Europe).

Additionally, ideologically it wasn't possible to appease Hitler. He wasn't looking for the best "deal" for the German state, he was looking for a new world order (something no appeasement could have ever given him).

Hopefully this answers your question, if I wasn't clear in my answer feel free to ask anything :)