One of the main reasons behind gunpowder infantry fighting in large blocks and lines is that they made the army easier to control, but surely lots of people must have foreseen the rise of firearm infantry, looked back at how badly tight, unarmoured blocks of people faired against ranged attacks, and put two and two together?
Prior to or during the rise of gunpowder, what attempts (if any) were made to advance battlefield communication between units so that these units could become smaller and more versatile, allowing individual soldiers to react more independently instead of standing there waiting to be shot?
Sorry if it's a bit long.
/u/BritainOpPlsNerf handled the particulars of battlefield communication so allow me to handle some issues I have with the question itself. I don't think this is at all a fair question. I know it's not exactly nice to criticize the question asked but this is really a completely unfair question to ask.
You assume people must have 'forseen' the rise of infantry being exclusively comprised of firearms. Why do you think this? Missile weaponry had always served an auxiliary role for the entirety of human history and even through the 17th century performed in that manner. Muskets were there to provide a shock factor not lead the army.
You seem to believe (at least from my interpretation) that death rates in this period were of some excess that was totally unacceptable and no one knew what to do about it. The fact is is that muskets were incredibly inaccurate and the only reason they replaced the pike as the principle weapon of war is the ring-bayonet being developed which allowed it to fire and act as a 'pike' simultaneously while also requiring little training to use. They were, in short, a utilitarian weapon that combined many previously complex jobs that would require a crap load of training into one where you can train someone to perform in a matter of weeks. However the fact of the matter is this: They were not horribly deadly and most people would not die just 'standing there being shot'. They would die when they ran and were cut down -- just like ancient warfare.
You also seem to imply that they were dumb for not using armor; asking why they did not "put two and two together". Armor was pointless as it did little to stop a musketball or a bayonet -- it was a cost benefit analysis. Armies could be significantly larger since it was easier to train a man with a musket and to fire when told than to perform complex pike formations or to have to physical strength and coordination to fire a bow in volley. By removing armor the average distance an army could march would double from about 6-7 miles a day to 14-16 almost overnight.
You're asking, in essence, for a Deus Ex Machina of history with your 20/20 hindsight. You're basically saying that "They had a problem where they were incapable of getting orders across a battlefield in rapid time so they used tight formations to rectify this. Why didn't they just invent radios/whatever to make it easier?" I don't mean to be crass but you can't just say why didnt people back then invent 'something' to fix a problem out of nowhere. It's totally unfair.
If I could have a brief rant for some mysterious reason people as a whole can 'get' why phalanxes were used but they can't understand why line formations were used simultaneously when you must understand they were acting under the precise same principle. Infantry acts as the anvil to hold the enemy in place while the decisive actor (cavalry, generally, but later reserved infantry charging in columns) targets a weak point and rolls the enemy up. When pikes stopped being so great in the face of musketry musketry took over but the exact same job needed to be done.
They were, especially in the 18th century with the rise of larger and larger armies, filled with people who could not be relied upon and had to be kept strictly regimented. The strength of musketry relied wholly in unit cohesion. The only reason muskets were so deadly to cavalry was because they could act as a mass similarly to a body of spears alone are vulnerable but together are a force. You can't just say that 'they didn't have radios therefore they used blobs until they were smart enough to invent new communication.' I know that's not what you said but it is the implication.
My issue, I suppose, is that you seem to feel like 'line infantry' was something that 'should' have been prevented. That's not a fair approach to history. Yes communication was a big factor but so was unit cohesion -- warfare first and foremost about making the enemy turn heel first and you did that by sticking together on your end and making the enemy break apart. Warfare in the period of the 'rise of the musket' was not particularly deadly. It was highly defensive and more centered on fortress based siege warfare and not giving the enemy battle when preferable. When battle did occur the extent of 'standing across from each other firing' was brief if at all. What was more likely to occur was one or two volleys being exchanged and then a bayonet charge to engage the enemy.
For the longest time skirmishers and light infantry were considered ancillary to line infantry and for good reason. They were just to vulnerable to the more concentrated firepower from line tactics, both in rifle and charge, and wouldn't have the weapons until about the late 18th and early 19th centuries to put them to use as a central component of an army. I really can not emphasize this enough. When we're talking about 18th century muskets, what is honestly more terrifying do you think? A couple of men being picked off in intermittent periods or 100 shots being shot at once in your direction. That's 100 bullets striking in your general region, the sound of 100 literal boomsticks going off, all that smoke, and then the roar of 100 men shoulder to shoulder charging in your direction with bayonets as you and your like 20 buddies in that small area look wide eyed at them coming right at you. You'd turn tail and run instantly. That is why they were used like this.
Further, coordination like this was the only reason coordinated fire and bayonet charges could ever work. Again, people have this thing against bayonet charges as being some relic from the past but they were incredibly effective and turned infantry into decisive actors. At the least no matter what if you performed a bayonet charge some kind of result would be determined then and there. Bayonet charges only worked with a tightly packed force acting as a unit and bayonet charges were what won and lost battles from the early 18th century all the way up until Waterloo. It was the weapon of warfare. It may be easier to think of muskets less as rifles replacing spears but more of them as just being spears that shot once or twice before engaging the enemy spears who shot a few times before engaging.
They didn't care about the factor of more men standing packed together will lose more to a volley than men spread out. They saw the benefit of putting largely poorly or untrained levies to their own devices and trying to think that they would have the capacity to perform small unit tactics. They didn't have the weaponry to perform these things, especially in the age of cavalry, nor did they have the inclination nor training to do so. They were better off being kept tightly together not only for communication but for discipline. It is a boost to morale when everyone is right next to each other backing each other up and you know you got a bayonet to left and right when that horse is charging right at you and it's easier for the officers to make sure you don't wander off 'accidentally' in a pitched battle.
EDIT: A good post by /u/DonaldFDraper on this topic. This excerpt puts it well:
The musket has several problems, first is the firing mechanism which depends on a piece of rock (flint) striking a piece of metal to make sparks. The flint can shatter in your face when hitting the metal but also it becomes dull and can contribute to misfires. Next there is a problem of gun powder which is not up to the quality of our current smokeless powder that wouldn't be invented till the later end of the 19th century. Next and most importantly is the problem of accuracy, due to the difficulty of rifling and reloading rifles, muskets were used that had bad accuracy at best, even under the best conditions a musket could hit a man sized target roughly 70-80% of the time, again under ideal conditions.
The real problem with the musket was the human aspect, particularly in reloading. A musket requires a charge in both the pan and the muzzle with the ball. Sometimes soldiers would have flashes in the pans, sometimes they might fire their ramrod, sometimes they might miss the musket barrel and pour the powder and/or ball in the grass or have the charge pour onto the ground because someone bumped you. Combine that with the fact that soldiers were trained not in marksmanship (which is rarely emphasized due to the limitations of the technology) and the focus on movement, a soldier fighting in the Napoleonic Era is very different than in the contemporary era.
Second, the tactics of the era are focused around the limitations of the weaponry. Because the weapons are inaccurate and limited, Napoleonic era combat was more focused on combined arms movement to ensure the defeat of the enemy. The French often focused more on the Steel (bayonet) rather than volley fire, combine this with quality cavalry and the best artillery in the world, French tactics were far more mobile and more focused on melee.
Further Reading:
The Rise of Modern Warfare: 1618-1815, H.W. Koch
Fire and Stone: The Science of Fortress Warfare 1660-1860, Christopher Duffy
The Age of Battles: The Quest for Decisive Warfare from Breitenfeld to Waterloo, Russel Weighly
Treatise on Partisan Warfare, Johann Ewald
Prior to or during the rise of gunpowder, what attempts (if any) were made to advance battlefield communication between units so that these units could become smaller and more versatile, allowing individual soldiers to react more independently instead of standing there waiting to be shot?
The earliest forms of communication, music and rhythm, became far more sophisticated, organized and militarized as time went on. Musical signalling - lets refer to it as that - reached its Zenith in the early to mid 1800s and was rapidly supplemented and expanded beyond the tactical level ('to the sounds of the guns!') to the strategic level in the 1850s and 60s with the ever growing density of rail lines and the telegraph movement. "Signaling" as we know it was used extensively by the Union army and allowed information to travel quickly across a battle line with incredible frontage with relative speed. The method involved using a series of flags to 'signal' (hence the name) information. This innovation is so important to the pre-radio battlefield that it remains the insignia of the US Army Singal Corps to this day.
During the Napoleonic Wars, Napoleon's genius at planning marches meant that lateral communication between columns could be kept to a minimum, and he used a very early 'telegraph' system based on a series of post stations (that moved with its parent unit) that had spare horses, and if need be a new Aide de Campe to take over and deliver a message. Combine this with multiple messangers to establish military redundancy and brilliantly pre-selected lines of communication, and orders for Corps, often miles away, would reach them in a timely enough manner that it legitimately saved Napoleon's ass on many occasions such as at the pictured Battle of Marengo.
We can fast forward to the 1870s, when military thinkers, especially the Prussian ones, began to realize that absolute command and control would be impossible with the technology of that time and began to develop and nurture initiative at the highest levels of command. The idea, loosely translated as 'mission oriented tactics' from aufgrastaktik stressed a commander on the spot knowing the objectives of a formation two pay-grades above his rank. So, in short, it meant that a well trained and educated Officer Corps wouldn't need immediate communication - they knew what to do, even in the vaguest terms, when they did not have orders. Or, perhaps even more importantly, even when they had contradictory orders.
The idea that command and control is suddenly very easy to maintain in a modern conflict is an over-used trope; even if we are speaking relatively. "The Fog of War" is still very much present on modern battlefields, even if it is no longer quite literally a cloud of gunpowder discharge cloaking a line battalion.
One last nitpick:
looked back at how badly tight, unarmoured blocks of people faired against ranged attacks, and put two and two together?
They performed astonishingly well until the late 1800s and spacing and practice changed and shifted considerably from the onset of Linear warfare to its quite hard to distinguish end. I refer you to that excellent rant/essay by /u/elos_ that will stress how they did not "Simply wait around to be shot."
As for why they weren't developed earlier: well because I'd argue that would be a serious case of putting the cart before the horse. Again, as always, speaking relatively, the rise of gunpowder is what prompted the need for greater C2; armies until that time had been small. They had been growing, yes, but they were still incredibly small compared to what was to come. Turns out, muskets don't take a great amount of training to fire - certainly less training then it takes to wield a pike in formation with a curiass on. We don't really see a levee en masse until 1792, in my view, but discounting that, the century of Linear warfare leading up to the French revolution still saw armies of a size and magnitude that on average rivaled even the greater battles of its preceding era's wars. One doesn't need some great leap forward in technology and doctrine to command an army of 12, 000. They certainly do however, for an army of 120, 000.
