The reason why coke changed it is a simple one, Pepsi happened. Whilst Pepsi had existed for practically the entirety of Cokes history they were never a large threat to the dominance of Coke after WWII. However this changed in the mid 70s with Pepsi running two well known ad campaigns. The first was the Pepsi Challenge with featured a blind taste test which showed that people preferred Pepsi to Coke. The second was an aggressive celebrity endorsement with the likes of Michael Jackson which pitched Pepsi as The choice of the new generation. As these campaigns were successful and the market share for Pepsi grew Coke knew they had to hit back. So they got food scientests to look into what was it that made people say the liked Pepsi and the answer they came up with was that Pepsi was sweeter then coke. So once they knew that they came up with a new sweeter version of Coke called New Coke that according to their own tests was preferred to both Pepsi and Old Coke.

Now as to why it was a failure there are two reasons that people give. The first is Nostalgia, Coke was a drink with a long history and very little had changed with it since it was first created. Coke underestimated just how much some people loved the original product and how upset they would be with old cokes complete removal.

The second reason is that the taste test that both companies ran were not accurate in reflecting what people liked. In all these tests only a small sample of the product was given not an amount you would actually drink. What this meant was that people would pick the sweeter drink even though if they were given a full can they might go with the other choice.

The third option is one I do not agree with but I do need to mention it. There are some who think that the whole new coke was a ploy by Coke to get the public to do its own ads after releasing new coke the airwaves were full of people singing the praises of old coke and when it was returned sales were higher then they were before New Coke.

The big problem with New Coke wasn't that it was an alternative, it literally replaced old Coke. You could no longer buy the old formula.

Coca-Cola drastically underestimated brand loyalty. Even if a new product is better, people like to go with what works and what they know. Taking away Coke completely and just saying 'This one is better' will make a lot of people mad and even outright refuse to drink it.

If the company had introduced New Coke as a separate drink, then over time popularity shifted towards it, they very possibly could have eventually stopped production of the old coke with no problem. But since they immediately stopped availability of the old formula, people were angry.

Wikipedia backing me up: http://en.wikipedia.org/wiki/New_Coke#Official_launch

When New Coke was launched, they stopped production of old coke within a week.

Some people even considered coke as part of their regions identity. Indeed, a lot of places in the south call all soda 'coke'.

http://www.marketing91.com/coca-cola-brand-failure/

http://www.nytimes.com/1985/10/18/business/old-coke-outselling-the-new-in-us.html

OP, check out this book!

http://www.amazon.com/gp/product/0140104089/

Even try reading the sample text. It's pretty interesting!

Follow up question: When did coke switch to High Fructose Corn Syrup (HFCS)? Why was no one outraged about that, since the difference in taste is quite noticeable?

TL;DR: CocaCola executives believed Pepsi marketing, and made new Coke.

Coke was specifically developed in response to the Pepsi Challenge. http://en.wikipedia.org/wiki/Pepsi_Challenge

In his best selling book "Blink" Author Malcom Gladwell concludes (correctly I believe) that Coca-cola executives were scared by losing short-term marketshare to the Pepsi challenge. Pepsi was making l gains in distribution/market share thanks to thanks to their "choice of a new generation" commercials along with hip marketing from both Michael J. Fox and Michael Jackson, etc.

CocaCola (classic) was superior to drink, but Pepsi which was sweeter would consistently win in a taste challenge of small cups. Pepsi would win the sip contest something like 1 out of 5 times, wereas CocaCola (classic) would win something like 7 out of 10 would prefer CocaCola (classic) in a can vs. can contest with Pepsi.
(That is because the human tounge gets overloaded by the sugar and needs the bitterness of CocaCola classic to balance it out).

I remember as a kid the Pepsi taste challenge happening everywhere -- like at grocery stores, etc. Coke Executives were terrified of losing their dynasty and decided to beat Pepsi at their own game.

Coke was sweeter, MUCH sweeter than Pepsi. They released Coke with much fan fare about how it beat Pepsi in Pepsi challenge.

So Coke would win a taste challenge, but by the end of the can it was too much sugar. Regular Coke drinkers were annoyed they had made an inferior product, and many switched to Pepsi as a result since it was (at the time) the less sugary of the Coke vs. Pepsi.

Ironically -- Pepsi wasn't actually making substantial distribution inroads until CocaCola released New Coke, that is what created the real surge in demand for Pepsi.

Coca-Cola changed to New Coke get back some of the people who had jumped ship to Pepsi, but they ignored the basic fact that while some people had switched to Pepsi, the majority of people still preferred Original Coke, and had the rug ripped out from under them when their beverage of choice-still the number one soft drink in the world at the time-was replaced.

It's similar to when Michael Jordan quit basketball to play Double A baseball, and then realized he was better off back on the basketball court.