This is called 'line tactics' or 'line formation' and was the most effective way for infantry to deal with the issues of warfare in their day.

The guns of the time were not very accurate and took a long time to reload and fire. By massing your troops into a line and having them fire in successive waves through the ranks while the other ranks reload (1st rank fires, 2nd rank fires, 3rd rank fires, 1st rank is finished reloading and fires again, etc.) you could dramatically increase the number of projectiles that hit the target.

The line also protected the soldiers from charges (whether from cavalry or other infantry). Skirmishers (soldiers that are more spread apart) are very vulnerable to charges, whereas massed infantry can reform into a square formation and decimate an incoming cavalry charge (the Allied square formations in the Battle of Waterloo fought off eleven cavalry charges).

Why did soldiers just stand there and take it? Well, they didn't always and that is often how battles were won. A lot of training went into the speed of reloading as well as maintaining the line in the face of casualties. If a side took too many casualties it often broke and fled and then could be easy pickings for a charge or for the cavalry to mop up.

The thing to keep in mind though, is that the line tactics weren't the only thing going on in the battle - they were a single tool to be used by the commander. There were also light infantry (skirmishers) who could act with greater speed and initiative, then hide behind the line infantry if they needed to. There was cavalry which could move quickly and attack from unsuspecting places. There was artillery being directed where it was needed.

Whole armies didn't really just march up to each other, line up and shoot until one side was dead.

I think the problem is that in fictional depictions, such as films, the amount of extras is limited, and geographical locations are very small. Even big sequences in high-budget films will only have a few hundred extras total, and take place over a couple of hundred metres (think the opening scene of The Patriot). At this smalls scale, yes, these lines seem very inefficient, compared to individual soldiers running about and firing at will.

However, in real life, these battles featured thousands of men (sometimes tens of thousands), and were spread out over huge areas. Battlefields were confusing places, and you really have to think about how you're actually going to organise and control even a single regiment of 500 men, let alone a whole army.

Fire-power in general was fairly limited in its ability to win a battle by itself. You also have to remember that all of your soldiers are likely terrified, and liable to panic if they feel isolated and exposed. Broadly speaking, you're not going to win the battle by killing everyone on the other side. You'll win by keeping your men on the field, under your control, firing controlled volleys as much as is possible, and hoping that the enemy breaks and runs first. Rigid formations allow you to make sure that your soldiers stay active and responsive, and maintain cohesion under stress.

So we're not talking about small battles and skirmishes, where individuals can make a real difference, Mel Gibson-style (these battles were often fought differently, and both the British and Americans were constantly developing tactics for these engagements). We're talking about enormous, terrifying, and above all confusing encounters, where the number-one danger is not the other side wiping you out, but your formations losing their cohesion and responsiveness. Whilst it might seem a little inefficient on a micro-level, on the macro-level it was the best way of ensuring that enormous armies remained effective and under your command.

(A great book for understanding the nature of combat in this era, and my main source, is Christopher Duffy's 'The Military Experience in the Age of Reason)

Yes, they did. The fact of the matter is if you have 100 men spread out across 600 feet and 100 men packed into 150 or so feet, the tighter pack of men are going to overwhelm every single group of the other 100 they come across. They will overwhelm them with more concentrated and coordinated fire because everywhere they go it will be 100 men vs like 10 to 30. That's really, ultimately, the core of why line tactics exist -- concentrating your fire and your men allows you to control their movements, for them to motivate each other and reduce chance of running away when they do get shot at and charged at. Like was said above, skirmisher and light infantry are more difficult to organize to repel cavalry charges. They may be superior in the sense that they can more easily take cover and "pick off" enemy infantry but they are, essentially, useless in warding off enemy cavalry and are only useful in 'picking off' the enemy standing in a line if they have their own line protecting them from being smashed into.

Firing a single volley and then charging with bayonets locked was not uncommon and 100 men crashing into 20 men who occupy the same area will crush them every time. For the longest time skirmishers and light infantry were considered ancillary to line infantry and for good reason. They were just to vulnerable to the more concentrated firepower from line tactics, both in rifle and charge, and wouldn't have the weapons or understanding of doctrine until about the late 18th and early 19th centuries to put them to use as a central component of an army.

With that said, light infantry was a huge component of colonial armies. When you see things like Last of the Mohicans with the British only standing in lines taking their time firing while Mel Gibson and friends dance around and take cover, that's not how it worked. The British loved to use light infantry in their colonies and used them extensively in counter-guerrilla tactics. Line tactics were certainly used and a central component to Western warfare but they were not the only form and were not used idiotically in areas where they were not effective.

The Napoleonic Wars (1792-1814, roughly) gave rise to a more central role of light infantry on the European continent, however. The Battle of Valmy was one of the first battles of the first war and was the really first battle showing the power of light infantry. The battle is noted mostly for its showing the professionalism of French artillerymen and was fought, primarily, as an artillery duel. However whenever the Prussians attempted to break the stalemate with their own infantry, French Tirailleurs would cause enough of a nuisance that the line infantry could not advance without unreasonable casualties -- holding them off. Although a technical draw (less than 500 people died out of a combined 66,000 men on each side) it began a period of the Prussian elite light infantry, the Jäger, began to show more prominence in the military. They would tend to operate in pairs and cover each other and never stray too far away from the line infantry. They also were capable of operating (relatively) independently and seize initiative, a pretty revolutionary freedom for infantry to be given in this period.

Every major nation would adopt or would professionalized light infantry brigades by the end of the Napoleonic Wars. The Prussians the Jäger, the French had Voltiguers and Chasseurs and Tirailleurs (all with slightly different designations and roles), the Austrians had Grenzers. Most of these, the Austrians in particular and the Portugese as well, were created originally from irregular groups of farmers (the Portugese word Caçadores literally translates to "Hunters") and would be gradually professionalized and organized into elite groups that men wanted to train into.

They were, like I said before, relatively independent and that required a level of self discipline and mental strength and more importantly situational awareness that required a lot of training and rightfully was designated elite status. These men were, ultimately, not a bunch of untrained irregulars but a bunch of highly trained and highly intelligent professional soldiers who may not have been the bone or the muscle of the army -- that was the line infantry and the artillery respectively. They were the sinew that bound the two together and kept them strong and flexible.

Well shit this turned into a massive post, guess I need to give a few citations.

Swords Around a Throne by John Elting

The Campaigns of Napoleon by David Chandler

The Wars of German Unification by William Carr

Gadarn's given a good answer as to what line tactics made a great deal of sense given the limitations of the day.

As for specific examples of this, there are quite a few. The first battle (or at least skirmish) of the Revolutionary War at Lexington in 1775 involved a short exchange of fire between militiamen and British troops. The 1776 Battle of Long Island involved a day-long battle that saw several exchanges of fire between troops in line (although the Americans got the worst of the the fighting). There's also the battles of Camden, Cowpens, Monmouth, Brandywine, Harlem Heights, and a host of other battles that saw both sides use line tactics.

With that said, the American Revolutionary War, like the French and Indian War, saw a great deal of unconventional tactics and formations being used. The Americans at King's Mountain or the Native Americans at Oriskany fought in loose order, rather than as tightly-packed units.

Very good comments already and not much to add but a very good book about warfare in this era is Redcoat by Richard Holmes.

One of the surprising things is the range at which volleys were fired. Sometimes fire would be witheld until an enemy was within 40 feet. The effect on morale of a well timed and well executed volley could be devastating. The battle of Quebec is famous for having been lost and won with a single volley.

Curious quest stemming from this one, where they many instances where the American militias / colonial armies took up covered positions that the British then proceeded to simply march towards? Al la Bunker Hill, or did the Americans attempt to fight the British line on line more often than not?