God I hate Reddit formatting. Sorry for the double post OP, RIP your inbox.

In movies and reenactments of the American War of Independence, soldiers are seen standing shoulder-to-shoulder in long rows facing the enemy. If I were designing a battle formation that would make it as easy as possible for my troops to get shot, that is the one I would pick.

Please don't take offense to this, but this is coming off as unbelievably smug. I know it's likely not intentional but it plays off the trope that people in the past were somehow stupider than us today with our infinite knowledge and it does peeve me quite a bit. Please understand that Generals were not stupid and they were not keen on throwing men away mindlessly like comic book villains. This is more of a general historiographical statement, I'll narrow down to your question in a moment, but if you're looking back at a historical idea or concept or practice and are thinking, "wow, that makes no sense at all -- why would they ever do that?" understand that you likely are missing something or are given the luxury of hindsight. It's very easy to fall into the trap of criticizing those before us for things that are seemingly obvious today.

However, that doesn't really apply to the situation here. Line formations were by no means stupid and the most effective way of waging war well into the 19th century. Let's explore this.

Was it in use from the beginning of firearm warfare? If not when did it become common?

Let me paint you a picture of the history of firearms up until the 18th century briefly. The very first muskets would be extremely heavy, lumbering, and not useless but not exactly amazing either. They would require a large stick with a "v" shape (like so) These would be integrated into the formation of the time, that being pike formations. The firearms would be mixed in with the pikes, using their firepower to decimate enemy formations and the pikes protect them. As you might imagine, it was effective enough that basically everybody started doing this. Naturally, musket design would improve over the decades. They would become light, light enough for a man to carry and raise and aim it himself. Design changes to the loading process made it more reliable in poor weather and increased the rate of fire from 1 shot a minute to 3 and maybe even 4 if you were good. And finally one major improvement would be added, the bayonet. This would start to become popular in about the mid 17th century.

If so, why was it preferred over something more spread out?

Let's cut to the core of the issue here, the fact of the matter is if you have 100 men spread out across 600 feet and 100 men packed into 150 or so feet, the tighter pack of men are going to overwhelm every single group of the other 100 they come across. They will overwhelm them with more concentrated and coordinated fire because everywhere they go it will be 100 men vs like 10 to 30. That's really, ultimately, the core of why line tactics exist -- concentrating your fire and your men allows you to control their movements, for them to motivate each other and reduce chance of running away when they do get shot at and charged at. Like was said above, skirmisher and light infantry are more difficult to organize to repel cavalry charges. They may be superior in the sense that they can more easily take cover and "pick off" enemy infantry but they are, essentially, useless in warding off enemy cavalry and are only useful in 'picking off' the enemy standing in a line if they have their own line protecting them from being smashed into.

Firing a single volley and then charging with bayonets locked was not uncommon and 100 men crashing into 20 men who occupy the same area will crush them every time. For the longest time skirmishers and light infantry were considered ancillary to line infantry and for good reason. They were just to vulnerable to the more concentrated firepower from line tactics, both in rifle and charge, and wouldn't have the weapons until about the late 18th and early 19th centuries to put them to use as a central component of an army. I really can not emphasize this enough. When we're talking about 18th century muskets, what is honestly more terrifying do you think? A couple of men being picked off in intermittent periods or 100 shots being shot at once in your direction. That's 100 bullets striking in your general region, the sound of 100 literal boomsticks going off, all that smoke, and then the roar of 100 men shoulder to shoulder charging in your direction with bayonets as you and your like 20 buddies in that small area look wide eyed at them coming right at you. You'd turn tail and run instantly.

With that said though, light infantry was a huge component of colonial armies. When you see things like Last of the Mohicans with the British only standing in lines taking their time firing while Mel Gibson and friends dance around and take cover, that's not how it worked. The British loved to use light infantry in their colonies and used them extensively in counter-guerrilla tactics. Line tactics were certainly used and a central component to Western warfare but they were not the only form and were not used idiotically in areas where they were not effective.

The Napoleonic Wars (1792-1814, roughly) is when we begin to see the rise of light infantry taking a front stage role in continental warfare. The Battle of Valmy was one of the first battles of the First Coalition War and was the really first battle showing the power of light infantry. The battle is noted mostly for its shocking show of professionalism from French artillerymen and was fought, primarily, as an artillery duel. Whenever the Prussians attempted to break the stalemate with their own infantry, French Tirailleurs would cause enough of a nuisance that the line infantry could not advance without unreasonable casualties -- holding them off. Although a technical draw (less than 500 people died out of a combined 66,000 men on each side) it began the period of the skirmisher. The French revolutionary armies would love skirmishers so much they would make pure skirmisher battalions -- to deadly effect. They would tend to operate in pairs and cover each other and never stray too far away from the line infantry. They also were capable of operating (relatively) independently and seize initiative, a pretty revolutionary freedom for infantry to be given in this period.

Every major nation would adopt or would professionalized light infantry brigades by the end of the Napoleonic Wars. The Prussians the Jäger, the French had Voltiguers and Chasseurs and Tirailleurs (all with slightly different designations and roles), the Austrians had Grenzers. Most of these, the Austrians in particular and the Portugese as well, were created originally from irregular groups of farmers (the Portugese word Caçadores literally translates to "Hunters") and would be gradually professionalized and organized into elite groups that men wanted to train into. They were, like I said before, relatively independent and that required a level of self discipline and mental strength and more importantly situational awareness that required a lot of training and rightfully was designated elite status. These men were, ultimately, not a bunch of untrained irregulars but a bunch of highly trained and highly intelligent professional soldiers who may not have been the bone or the muscle of the army -- that was the line infantry and the artillery respectively. They were the sinew that bound the two together and kept them strong and flexible.

When did it fall out of favor?

1870 is about when. The Franco-Prussian War taught a story to Europe that many would not want to hear, but would harken in an age of new warfare. As opposed to the ACW just five years prior which used muzzle loaded muskets, the French and German forces would both be using breech loaded bolt action rifles using cartridges. The French had the Chassepot and the Germans had their infamous "Needle Gun" -- both with an effective range over a thousand meters. I'll quote from Michael Howard:^[1]

The German infantry did not, indeed, acquit themselves particularly well. The company columns in which they advanced into action disintegrated under fire into a ragged skirmishing line which quickly went to [the] ground, and which officers and N.C.O.s urged forward in vain. In the woods and close country which lay before the French positions the temptation to 'get lost' was sometimes overwhelming. Only close order could give the infantry confidence, and close order in the face of breech-loading rifles was suicidal. The answer to the problem, as the Germans discovered during hte course of the campaign, was for the infantry, so long as its armament was inferior to that of the enemy, to hold back and leave matters to the guns; and the German field artillery proved quite capable of settling matter sitself. Its range and rate of fire gave it, at the beginning of both battles, such an ascendancy that the French gunners -- including the dreaded mitrailleuses--were silenced in a matter of minutes.

The Franco-Prussian War was a "half and half" war even more than the ACW. The Germans would have rapid mobilization -- over 250,000 men -- and would have staggering casualty rates. They would simply not be capable of assaulting positions without unacceptable casualties because of the deadliness of French riflemen and them not having the tactical flexibility to deal with it. The Generals had no idea what to do other than to just sit back and try and flatten the target area with their artillery and send in their infantry to mop up -- something we'll see tried again in a few years with much less success. However it worked now and, unfortunately, both sides didn't get a real picture of the futility of their tactics because of how much of a fluke the war was. You could argue it persisted into WWI briefly but that's a stretch. Ultimately the line tactics you know and love would die in 1871.

---

Notes:

^[1] Howard, Michael The Franco-Prussian War: The German Invasion of France 1870-1871, pp. 118

Elting, John, Swords Around a Throne: Napoleons Grand Armee

I think it's best that I have a master post for me to reference for this.

First, I'll address your criticism of the formation. The Charleville Model 1777 (which was used by Americans due to French supply) had a generally 83% accuracy rating in a perfect field with perfect weather at fifty meters as the person using the musket had all the time in the world to reload, aim, take their time to understand how the gun might fire, compensate, and fire. SO, that might take two or more minutes. Under the same conditions, a Brown Bess India Pattern had a roughly 76% accuracy ratio. However, in the heat of battle, misfires were common and accuracy couldn't be ensured so commanders had to rely on accuracy by volume. Even a rifle in this time period would take more than a minute to reload and were terribly expensive to make compared to a musket. I apologize but to understand the time period, you must understand the weaponry and why they chose to use line formations.

There is two ideas in the 18th century of how an infantry battalion should fight: line and column formation, both with advantages and disadvantages.

The Line formation is the standard formation of the 18th century and is the ideal for what you described, which is called a Line Battle. In a line battle, discipline and consistency of fire was ideal. In the 18th century, line infantry were trained to fire and reload their muskets with precision and rapidity that they could fire two to three times per minute in order to make the enemy line melt away. The reason why a line was used was to maximize fire power. Ideally, a battalion in line formation would be two to three men deep in order to maximize the number of men firing, anyone behind the third line would be unable to fire due to the men in front.

However, during a campaign, movement was via column which was men , usually five to ten men wide and as deep as needed, the column lacked fire power but had shock. Although the column wasn't used as much in the 18th century as the line formation, it was used for a bayonet attack as it could use sheer weight of men to crash into a single point.

Another reason why spread out formations wasn't used was due to the difficulty of commanding troops that weren't stuffed together. Up until the French Revolution, most armies tended to use the dregs of society in the line with nobility in the officer slots. As a result, it was believed that soldiers were too dumb and too unruly to be allowed to step out of line, and they would need to stay in formation to even be effective. After the Seven Years War, nations started to experiment with open order fighting/skirmishing but France took it to the next step during the Revolution thanks to the sans-coulettes and the need for soldiers to fight.

The line formation started to become common at the end of the 17th and the beginning of the 18th century when Europe generally started to use only firearms for their armies. However, the line formation fell out of favor around the end of the 19th century due to accurate artillery and cheap production of repeating rifles.

I hope that this helped you but you must consider that the weapons are inaccurate at best. Other things to consider are the fact that men side to side make it difficult to reload properly, not all gunpowder is perfect and could be a dud simply from non-industrial and non-perfected gunpowder factories, flits could shatter and hit someone, and finally a musket is a horribly heavy weapon due to the wood. An eight pound musket feels much heavier than a twelve pound modern rifle, simply due to the weight distribution.