It's important for us to separate those very rare occasions where a bayonet charge broke a modern force with machine guns and remember that before the development of the bolt action rifle, the bayonet and the musket had a parity of force at a certain distance.

This comes from David G. Chandler's capacious "Campaigns of Napoleon," which I recommend you read. Here you can see WHY bayonets became part of a standard soldier's kit, and why also they could be effective weapons:

"At the very best the musket was an inaccurate weapon. Toward the close of the eighteenth-century, the Prussian army conducted some field firing experiments with their own musket which differed little from its French counterpart in terms of accuracy. After setting up a canvas target 100 feet long by 6 feet high to simulate an enemy unit, they drew up a battalion of line infantry at varying ranges and ordered the men to fire a volley. At 225 yards distance, only 25 per cent of the shots fired hit the target; at 150 yards the proportion was 40 per cent; at 75 yards, 60 per cent of the shots told. It would appear therefore that the number of casualties caused by deliberately aimed rounds fired at normal range were comparatively slight; most troops were killed or wounded by artillery fire or “casual” bullets. However, when fire was held until the troops could literally see “the whites of the enemy’s eyes”— say 50 yards— horrific casualties could result, often as many as 50 per cent of a unit’s personnel— or even more. Thus at Austerlitz the 36th Regiment lost 220 out of 230 of its grenadiers, while at Auerstadt (1806) Gudin’s division lost no less than 124 officers and 3,500 men killed and wounded out of a total of 5,000 men committed to action.

Little wonder, then, that many French experts advocated shock-action with the bayonet rather than fire-action with powder and ball in the hope of avoiding such crippling casualties in a straightforward stand-up fire fight. Every French infantryman was armed with a 15-inch triangular-shaped socket bayonet. The real effectiveness of this sidearm is also open to doubt, however. It would seem from the result of researches conducted by Surgeon General Larrey that the havoc wrought by “cold steel” was more moral than material; after studying the casualties— French and Austrian— in two separate units that had been engaged in a hand-to-hand mélée, he could discover only five wounds directly attributable to the bayonet, whereas wounds inflicted by bullets during the same actions numbered 119.21 As a general rule, therefore, it was fear of the bayonet rather than its actual application that caused units to turn and flee. Perhaps this was all for the best, for the quality of the French bayonet was far from good. . ."

So that was a lot. What's the takeaway? The takeaway is this - in most combat, the force of a charge is more than just physical, it is psychological. Think about it: if you're in the front rank, and one of those stickers goes into you, you're very likely to be seriously hurt, and maybe even dead. You don't want to die. You've been shooting at these guys for how long, and they keep coming, and you can't seem to get them all. So, you break order, even if you're not fleeing to the rear, to try and steel yourself to the charge. But this bit of movement inspires others toward moving too. Suddenly the guy next to you thinks you're trying to bolt. Well, he won't stand and die if you aren't willing to - so he bolts. And then the guy next to him looks over and sees two men appearing to flee in the front rank. And then we're off to the races.

It's important to remember also that in the time of linear infantry tactics, breaking and running was a common, common thing in combat. Generals were prepared to endure troops whose courage was not absolute, because the task demanded of the common soldier was truly brutal.

Check out this description from Keegan's masterful Face of Battle. Here he is describing the majority of wounds endured by soldiers in the battle of Waterloo:

"In the infantry regiments, the majority of the wounds suffered were by missiles. Cannon-ball wounds were by far the worst; they took off heads, killed and wounded several men in line, killed a man and his horse; to be hit by a cannon-ball, unless in the limbs, almost certainly meant death. Grapeshot ranked next in lethality, but were not necessarily lethal: Lieutenant Doherty was ‘struck by a grapeshot in the stomach, and instantly afterwards by a musket ball through the head’ but lived to write of it in 1834. Musket balls, though much the lightest of missiles, killed easily: Canning, one of the Duke’s A.D.C.s, ‘received a musket shot in the centre of the abdomen, and, although perfectly collected, could hardly articulate from pain… Raised to a sitting position by placing knapsacks around him… a few minutes terminated this existence.’ Because of the low velocity of musket-balls, men could be seriously wounded by them without being knocked over. Hence the reports we have of wounds suddenly appearing on the body of whoever the reporter was talking to or looking at: Dawson Kelly was talking to General Halkett when the latter ‘received a wound in the face, the ball passing through his mouth’ and Hamilton, of the Greys, caught sight of a Royal Dragoon ‘whose cheek, just as I looked at it, opened, while I felt a ball pass close to my lips.’"

So you begin to see the horror of fighting in a line. You stand - sometimes for hours - in a fixed, uncomfortable formation, wearing a lot of heavy equipment, carrying a very heavy, cumbersome gun. You're cannonaded, sometimes for hours. And then you face a charge. You've been whittled down some already - and now here are these men, who seem pretty fresh, coming at you with sharpened metal stickers on the ends of their rifles. It took more than courage to withstand a charge. In many ways, it took a willingness to die no matter what the cost.

Psychologically speaking, nothing prods better than a sharp blade.

I'm cross-posting this answer from r/militaryhistory because I figured this sub would enjoy some resources/another perspective expounding a little bit on u/HMSErebus' answer.

I'm not sure the incident you're referring to with the USMC, but the last major bayonet charge took place in Iraq, and was done by Scottish units in 2004. Generally, it's considered that the enemy ran away because they were poorly-disciplined militia, and because the idea of suddenly being skewered by a burly Scottish infantryman screaming at the top of his lungs is equal parts shocking and unappealing. Now, to address the other points of your post:

During the Napoleonic Wars, this was not only a common tactic of Napoleon's, but the preferred tactic of every army for two reasons: A. It's truly frightening to see a wall of men rushing at you with no sign of breaking, and even worse when they've managed to flank you and are coming at you from the side. In that case, you only have however many ranks deep your line is to use against them, rather than your entire line's firepower. B. The musket of the time was, while more accurate in skilled hands than many people believe, still very difficult to aim effectively--there's a Ph.D. re-enactor whose name escapes me at the moment who showed that with sufficient practice a relatively tight shot-grouping could be achieved, as well as reload rates of up 5 rounds/minute or so, but the average soldier did not practice this much. Thus, a bayonet charge gave a greater chance of killing the enemy. The Red Badge of Courage, meanwhile, is supposed to portray a coming of age in battle, and thus it makes sense that the hero's green regiment would break and run.

Your comments on the Americans fighting in WWII are misleading. Proportionally, very few American troops were professional soldiers, most were draftees or volunteers who had signed up specifically for the war. There's a major difference between the man who signs on knowing he's going to be asked to kill the enemy for the rest of his life and the man who signs on to serve for a few years. However, there are several other factors you're ignoring here. As S.L.A. Marshall pointed out, only 15-20% of American soldiers in WWII would actively engage in combat--the rest were usually attending to "vital" tasks other than firing. LTC Dave Grossman, in his book On Killing, suggests that this is because men in combat don't like killing other people, especially at close range. Grossman also argues that in an animalistic "fight or flight" scenario like combat, the assumption is that the loudest "animal" will win, and the "animals" on the receiving end of that noise will automatically run. Finally, keep in mind that the Banzai charges by Japanese soldiers were predicated in part on a suicidal mentality--they assumed they were dead, and would take it as a bonus if they lived through the experience.

To answer your final question, video games and movies are not reality. The reality is that it takes practice defending against bayonet charges in training to defend against bayonet charges in the field, and most insurgents--for that matter, many militaries--don't practice this routinely. It's also very difficult to hit a moving target, even one moving towards you, and even more difficult to control the fear that will hit you when you see a 6' 200lb man + all his armor and LBE running towards you.

For more information, read Grossman's On Killing.

Edit: formatting

One thing I forgot to mention here - you might be facing a problem regarding selection bias. I have heard tell of occasionally, very successful bayonet charges into the modern era. Perhaps someone could write a book on that topic. But I haven't heard any exceptional stories of failed bayonet charges (with the exception, possibly, of the lancers apocryphally charging Panzers in Poland)- they'd likely be folded into all the other stories of attackers being gunned down by machine guns.

What I'm saying is, you're not hearing about all those other bayonet charges that didn't work - only the ones that succeeded "against all odds."

First, I suggest taking a look at this site:

http://www.napolun.com/mirror/napoleonistyka.atspace.com/infantry_tactics_4.htm

Bayonet charges sending entire armies into flight were in fact largely a myth. Jomini, one of the generals who actually witnessed Napoleonic era fighting, claimed to have never witnessed a charge larger than battalion strength; and that one side or another always broke before actual hand-to-hand combat could take place.

The reality instead is that the bayonet charge was supposed to be culmination of cumulative losses and stresses imposed upon the defender; leading to the collapse of his morale. The Napoleonic French for instance relied heavily on their skirmishers to inflict casualties on a defending line before finally charging. The defenders, who had seen many men (and possibly most of their officers) fall to a handful of skirmishers would thus be intimidated into running by a huge mass of men.

This mentality persists to the modern day. "Professional" and "Well-trained" does not mean that they stop being human. Being scared and wanting to run away is the norm, not the exception in combat. Indeed I would question the idea that any US infantry forces in World War 2 could be considered such when the vast majority of the infantry was conscripted.

Follow up question..... I'm new here and don't know if this is allowed..... But it's something I've often wondered. I remember seeing an episode of happy days as a child where Richie becomes a conscientious objector after undergoing bayonet training and realizing he couldn't be a killer. How did they train troops throughout war to have the discipline to charge across a field and stab someone to death..... How has military training encorporated the need to create killers into its training. Also note I have the utmost respect for our military and their ability and desire to do a job I would never myself be mentally or physically prepared for.