Part of this is that soldiers spend far more of their time marching than fighting, and so a common trend through all of military history is that soldiers will just drop things they think aren't useful, or are too clunky or heavy. Military theorists in the 16th and 17th centuries complained that soldiers would even hack off significant portions of their pikes to make them easier to carry, which would surely put them at a disadvantage when they went into battle. Yet they did it anyway.

It should be said that musketeers - however you'd like to define them or what you'd like to call them - often did wear armor. Breastplates were fairly common in the 16th and 17th centuries for firelock armed infantry, and armor remained a piece of equipment often required for service in European and colonial militias. Excavations at Jamestown have found quite a few breastplates modified to make using firearms easier, and we even know that the governor ordered in 1611 that even men bearing muskets had to wear armor when mustered.

The period you're specifying here is vast and there were enormous changes between 1571 and 1862 in how armies were raised, paid, fed, led, armed, equipped, and organized. There were so many changes that the period between roughly 1400 and 1700 has been called "The Military Revolution" and has been subject of a great many debates between historians about the mechanisms of change and whether or not there was a revolution at all. I bring this up mostly to point out that there was no single cohesive system of raising and fighting armies in this period, at all, though as states exerted more control of their armies they tended toward national levels of cohesion and organization, but whats true of British armies isn't necessarily true of British colonial armies, to say nothing of their difference to the French, or Spanish, or various states within what we now call Italy or Germany. While most states were trying to solve the same problems - largely logistics and discipline issues rather than technological or tactical - they went about solving them in different ways which manifested in very different organizational and doctrinal paradigms.

Numerous technological changes did influence a lot of these broader changes, though. Experimental ignition systems led to the widespread adoption of the flintlock, a larger and more experienced gunpowder production industry meant larger amounts of much more stable and powerful powder, larger scale production processes for building and storing firelocks meant that larger armies could be equipped more readily, and the introduction of the bayonet meant that firelock armed infantry no longer needed to be protected by pikemen, and by 1862 rifled muskets were very common, thanks to the Minié ball. But I want to stress that the larger differences weren't in tactics, they were in organization, supply, leadership, and structure.

All that established, why did plate armor stop being commonly used? Partially, it's because as states exerted more control of their armies, they also took on a larger burden of equipping and uniforming them to a consistent standard. While often individual regiments were responsible for their own initial equippage, they did so using, essentially, approved bits of kit and a "standard" firelock, though that too varied quite a bit. Most of the time, armies were raised at need, and so the speed of their muster and outfitting was important. Breastplates take a long time to make in bulk and add another burden onto the already-strained supply apparatus. While certain units - often cavalry units - still wore armor, it was extremely uncommon for infantry to wear any at all by the 18th century.

Part of this does reflect battlefield conditions. Even though the bayonet allowed a soldier to, theoretically, fight in a melee, that's not often how it actually worked. Bayonet charges tested morale and cohesion more than melee ability, and one side or another would break and run or come to a staggered halt before actually reaching the enemy line. Winfield Scott, reflecting on a bayonet charge into the British flank that won the 1813 Battle of Chippawa, said that "it is not in human nature that a conflict like that should last many seconds." Other observers of combat at the time described the effect of bayonet charges as turning lines into "ropes of sand" that simply blew away in the face of advancing bayonets. Nobody wants to fight with a bayonet if they could possibly avoid it, and so they avoided it when they could, exceptions largely being when men were trapped in an obstacle or defending a fixed position. Bayonet fencing training was often oriented more at giving men more confidence in their skills rather than actually expecting them to fence against their enemies on the battlefield. It does, however, argue that the musket or rifle itself was considered enough to practically defend oneself inthat situation; having a buckler or secondary weapon would just be more weight with no real utility.

In the sense that bayonet charges were unlikely to lead to prolongued hand-to-hand engagements, a breastplate is much more likely to slow you down, either on your charge or on your withdrawal. This doesn't mean melee combat never happened, just that the nature of it meant that a breastplate wasn't likely to make a great deal of difference.

There is an interesting and quit short-lived exception, though. During the Peninsula Campaign in the American Civil War, rebel soldiers were surprised to find several dead federal soldiers after the Seven Days battles wearing breastplates. Some observers noticed that several had dents made by rifle bullets, proving that in at least that sense they were useful, but a canister round had sliced through it where the rifle bullets hadn't. And since the breastplate only face forward, it was of little use in a retreat. These were private-purchase breastplates and were never standard issue. They appear to have more or less disappeared after the Peninsula Campaign, both on account of their expense, weight, and perceived utility, as well as the ridicule that they drew from fellow soldiers and enemies. That last point is important: though we like to think of military campaigns as rational exercises in scientific planning and machinelike precision, perceptions of masculinity and ideas of manliness or honor manifest in ways that might seem, to a modern eye, as irrational, inefficient, or unintelligent.

The tl;dr here is that breastplates and other armor continued to be used variously over the range of the period you've specified, but its utility as muskets and rifles began to dominate battlefields was dubious, the weight of carrying and wearing them was burdensome, and they were too clunky and expensive for thrift-minded states to purchase in large quantities or for men to purchase individually. Even if armor was effective, wearing it when others didn't might lead to ridicule, and would be an added burden on long marches and in poor weather.

---

If you're interested in this period I'd say one of the foundational texts you should look to is Geoffrey Parkers The Military Revolution: Military Innovation and the Rise of the West, 1500–1800 which is the preliminary modern text on the theory.

Afterward, I'd recommend reading The Military Revolution Debate: Readings On The Military Transformation Of Early Modern Europe edited by Clifford Rogers for some counterpoints and additions to Parker.

The debate is complex and periodized, but Parker and the essay collection will point you to further reading in areas you're more specifically interested in.

"why did footmen not wear something that could turn a bayonet aside?"

Because contrary to popular belief most bayonet charges didn't actually end with people being stabbed, they ended with one side giving ground. Preferably by retreating in good order, but sometimes by just running away.

Joseph Bloomenfield, a rebel officer at the battle of Brandywine in 1777 writes of the engagement that "we broke and rallied and rallied and broke from height to height." He describes repeated instances of being charged by British bayonets, running away, rallying, then doing it again.

This fits extremely well with the basic thesis of Matthew Spring's book, "With Zeal and bayonets only." He describes the British army using the bayonet to drive rebel forces to flee repeatedly, with instances of the bayonet charges resulting in actual stabbing largely limited to engagements like Paoli, where the rebels were caught unaware so they were unable to flee, or engagements like fort Lee and fort Washington, where one side was defending an entrenched or fixed position.

Spring's book also includes a lot of discussion on the value of troops remaining light on their feet in order to efficiently avoid getting stabbed, and to press home a charge/chase, which is very difficult to do when heavily encumbered. British troops are ditching as much weight as they possibly can in order to remain light, with examples of highland troops and grenadiers placing their swords into storage when actually on campaign, because who needs a sword when you've got a bayonet? One hessian soldier, I think Johann Dhola but I could be wrong, penned an interesting observation that British troops were incredibly light on their feet so as to move quickly, while their hessian counterparts carried more extraneous stuff and were not as quick as a result.

All this is to say that the protection offered by armor, which would only work against one threat that isn't the primary casualty producer on the battlefield, is outweighed by the reduction in tactical mobility and added cost.

You may be able to find more info in Spring's book, or maybe Christopher Duffy's "The military experience in the age of reason."

If we jump back a century form the invention and widespread use of the socket bayonet to the age of pike and shot warfare, there is one really interesting situation where firearm equipped infantry did take to wearing armor: Early Jamestown Virginia colonists fighting the Powhatan.

The archaeological reports from the Jamestown excavation include quite a bit of armor, but one really cool piece is a breastplate with a flange added near the right armscye that makes it much easier to shoulder a musket while wearing armor (a task which in unmodified armor, is highly unergonomic and makes good cheek and shoulder weld difficult.) They also found pieces of chainmail during the Jamestown dig, which doesn't present the same shoulder weld issues as a piece of plate. So what's up with that? Why at Jamestown are the modifying armor to be used with guns, and issuing hugely outdated armor to their gun armed troops in a way they did not in Europe? Look at the arms of their enemy. The Powhatan were fighting with stone age tech in the early Anglo-Powhatan conflicts that could be reliably stopped by modified pike armor, or by old mail. And since pike blocks proved useless against the Powhatan, just give EVERYONE a gun and some armor. Once native people gain reliable access to firearms, the armor ceases to be worth the added cost and loss of mobility, and so it quickly was abandoned.

"Some bring in a custome of too much curiositie in arming Hargabusiers, for besides a Peece, flask, Tutch boxe, Rapier and Dagger: they load them with a heauie Shirt of Male, and a Burganet: so that by that time they haue marched in the heat of the Sommer or deepe of the Winter ten or twelue English miles, they are more apt to rest, thē readie to fight, whereby it comes to passe that either the enterprise they go about, which requires celerity, shall become frustrate by reason of the staie they make in refreshing themselues, or else they are in daunger to be repulsed for want of lustines, breath, and agilitie.

Wherefore in mine opnion it is not necessarie, that this extraordinarie arming of Shot should bee vsed, but in surprises of Townes, Escalades, and assaultes of breaches, to defende the Souldiers heades from stones, and such stuffe as they besieged haue prepared to driue them from their enterprise: Or else in some speciall set battaile against the cut and thrust of Weapons, which exploits, for that they bee not so ordinarie as is the Skirmish, so are these armes nothing so necessarie, but rather a burthen more beautifull then beneficiall, and of greater charge then cōmoditie, specially a shirt of Male, which is very dangerous for shot, if a number of those small peeces should be driuen into a mans body by a bullet." - William Garrard, 1587

"The ordinaries should be armed with a Corslet in manner as is abouesaid, and should moreouer carry a Target at their backs, wherwith they might help themselues after that they are come so néere vnto the enemy that the Pike could do them no more seruice, they might therewith also couer themselues from Bowmen and Crossebowes, & at assaults, for asmuch as ye Pike is there a thing almost vnprofitable. And my saying must not be thought to be strange, for that I lade these men with so many kinds of harnesse, for I séeke but to arme thē surely, as men that must tarry by it, ought to be armed, and not like vnto those that arme themselues lightly; who being ill ar∣med, do thinke rather to runne away then to vanquish. I take also mine example from the Romanes, who armed their soul∣diers which they appointed for their battailes, as heauy as they might possible to make them to stand the surer against their e∣nemies, & that féeling their bodies so laden with harnesse, they should not hope to saue themselues by flight, but to dye in the place, or to win the victory. Vegetius complaineth yt the Soul∣diers in his time were too light armed, and followed not the aun∣cient Romanes, who commonly did surpasse & vanquish their enemies, because they were alwaies well armed, and the others ill armed. If our souldiers will then be accompted for to be va∣lianter men then their neighbors, it is necessary that they should arme themselues as sure as they might possibly, chiefely those that should be the force of the battailes, and so likewise should ye others that are for skirmishes, to giue their enemies the more trouble to defend themselues, and to be of the more force to re∣sist them. And for this intent I say that the extraordinary Piks should be armed with curets, sléeues of male, and with a good headpéece. The halbards should be armed likewise after ye same manner: and the Harquebusiers, Archers, and crossebowmen should be armed with a shirt & sléeues of male, and with a good headpeece: or for want of a shirt of male, they should haue cotes of plate, and good Iacks, yet they are almost out of season, but that maketh no matter, so there be any aduantage to be found" - William de Bellay, 1543(? I think its 1543, I dont remember if this was for the English translation or if it was when William himself published it)

From this we see two forms of thought: To make your musketeers wear armor, in case they fight in hand to hand, or to make them wear no armor to better aid their stamina in the skirmish. Both are valid thoughts, but one seemed to definitely win over the other.

With the fact that the armor worn by the common soldiers (unproofed armor) would be able to be penetrated at some 200 meters, and that they would take cover in the many skirmishes (be it entrenchments, trees, rocks, houses, etc), the need for armor doesn't seem so great.

​

However, saying all of this, it would be untrue to say that hand to hand combat would be uncommon. Trench raids have their debut from at least the end of the 15th century, and Blaize de Montluc's commentaries disprove the idea that handgunners would only be shooting.

​

By the Napoleonic Wars, bayonet training would be very little iirc, and whilst I am sure many trained on the use of it by the command of their officers, it would not have been standard until the end of the 19th century. Likewise, bucklers would not be useful when using the bayonet, and you turn the bayonet away with your own bayonet. However, I suspect that in the 16th century many men did carry bucklers, although not all, following the tradition of the archers.