This is my first answer here, so I'll do my best to make sure I don't break any rules or provide bad information.

In order to understand those massive infantry formations like in the picture you linked, you need to understand the weaponry involved. The muskets carried by soldiers during the period of line infantry were usually smoothbore weapons. Meaning there was no rifling inside the barrel. As you may or may not know, rifling is a set of grooves that wrap around the inside of a barrel to put a spin on a projectile. This leads to much more accurate fire. Lacking this, the projectile fired from a musket (the musketball) would bounce around the inside of the barrel before it exited, with the result of horrendous accuracy even as close as 200 paces.

Large mass formation, like what you linked, essentially allow the law of averages to come into play. These formations would fire in volleys, shooting more or less a wall of lead in the direction of their enemies.

From Flintlock to Rifle: Infantry Tactics, 1740-1866 provides a pretty comprehensive breakdown on some of the formation and fire methods.

The mid-19th century saw the development of the minie ball, so named for it's inventor, Claude-Étienne Minié. This was a much more accurate projectile fired from a front-loaded rifle, and it had a tendency to flatten on impact with a human body. This made it a nightmare for battlefield surgeons and accounted for the enormous pecentage of amputations during the American Civil war.

Rifled muskets were also an imperative to revamp infantry tactics. Closed formations in the face of accurate firepower is a recipe for disaster as evidenced by the casualty rate of the American Civil War.

The previous passage I linked also mentions the development of advanced visual and telegraphic signals. Coupled with the increased range and accuracy of artillery, this is one of the most important advances toward modern warfare. Rapid communication allows this artillery to be fired at an enemy that is not within the line of sight of the artillery technicians by using spotters who could relay information back to the gunners.

Artillery in its form during the Napoleonic Wars could be devastating enough when used properly, such as in enfilade fire (down the side of a formation), but more accurate and explosive artillery fired from an enemy the infantry could not even see would make easy work of those large formations.

Edit: Changed "death toll" to "casualty rate."

This change happened in the 19th century European wars.

u/chikindiner mentions the importance of rifles replacing smoothbore muskets. This was a factor. But rifles did not end the use of massed infantry formations.

In the Franco-Austrian war in 1859, the Austrians had modern, long-ranged muzzle-loading rifles. They faced the French army of Napoleon III, who favoured rapid bayonet charges in dense columns. The Austrians deployed in lines.

The French won their war handily, despite the firepower of the Austrian rifles. This probably had more to do with the poor training of the Austrian conscripts, but it served to confuse observers in Europe to the power of the rifle. Indeed, their experiences in this war led the Austrians to change their tactics from firing lines to bayonet-charges in column.

Meanwhile, armies like the French and the British continued to deploy their armies in lines after 1866, but compact, narrow-fronted lines with deep reserves.

It was the Prussians under Helmut von Moltke who truly changed things up.

Firstly, he decided to equip the Prussian army with the breech-loading Dreyse Needle rifle which could fire five rounds per minute to a Minié ball rifle's 1. Other European armies had rejected such weapons because... well, they were a new invention and rather unreliable. Indeed, during the war between Austria and Prussia against Denmark, Austrian observers noticed that many Prussian soldiers fired their rifles from the hip because they were too nervous to place the rifle near their faces. (Gas could escape from the breech when firing, which could burn the soldier.) Fire discipline was also an issue, with formations running out of ammo too quickly due to their weapon's high rate of fire.

Therefore, secondly, Moltke introduced much greater peacetime marksmanship practice for his conscripts. To illustrate: an Austrian soldier in the 1860s got 18 practice rounds per year. A Prussian got 100, and had to keep a log of each of his shots in his schiessbuch

Thirdly, and finally, the Prussians abandoned line-tactics. Instead of massed line formations, he made companies or even platoons the standard operating units of his army, and authorised them to advance and manoeuvre on their own initiative, as long as they were still acting in accordance with the overall battle-plan. This was revolutionary, and allowed the well-trained Prussian soldiers to use their rapid-firing rifles to amazing effect.

In the Austro-Prussian war that followed in 1866, these weapons and tactics allowed the Prussians to achieve a 4 to 1 casualty ratio in their advantage, even when attacking a fortified Austrian position across open ground at Skalice, and even though the Austrians by this time had much better artillery than the Prussians.

And yet, this was not immediately followed by other European powers. A French officer, writing in 1868, reported that although the Prussian tactics enhanced their firepower, it made the Prussian lines chaotic and uncontrollable, and that a more competent opponent than the Austrians would have overwhelmed them with a massed counterattack. (The French were mistaken. They did not realise that although the Prussian formation looked chaotic, in fact all the company commanders had been fully briefed on the battle plan and were cooperating.)

The French, instead, maintained line tactics, but added trenches. They also introduced modern rifles, and trained their soldiers to be excellent marksmen. In the Franco-Prussian war, the Prussians captured French rifles and discovered that they were sighted for 1500 yards. Few modern riflemen are effective at such ranges. They also fortified their positions with early machineguns and artillery. The French reasoned that their trench-lines would mow down the Prussian swarm-formations.

They were not completely wrong. In the Franco-Prussian wars, the French with their now more advanced rifles inflicted staggeringly heavy casualties on the Prussians. Indeed, the Prussians in the end resorted to forming in columns to charge rapidly at the French, only spreading out into their small units once they got in range of their older needle rifles. The Prussians still won the war, because their flexible formations allowed them to outflank French positions more easily, because they by now had extremely good artillery, and because they vastly outnumbered the French army thanks to general conscription.

After 1870, all European powers adopted the Prussian model of offensive, mass-infantry-based warfare. Only defensive firepower increased even further from this point on, with the introduction of repeating magazine-fed rifles in the 1880s, as well as more advanced, truly automatic-firing machineguns, and even better artillery. This would eventually lead to the massive slaughters in the first battles of World War 1, as everyone practised a completely offensive doctrine. They would learn, of course. Small unit tactics would be much further refined throughout the war. But that's getting rather far afield and would be the subject of an entire different series of posts. (That I wouldn't be able to write.)

So to summarise: More accurate, rapid firing infantry guns, machine guns, and artillery gradually rendered line-tactics non-viable over the course of the 19th century. But this lesson was learned by different nations at different paces, people sometimes drew the wrong conclusions from the effects they observed, and the development was not so neat and linear as one might assume. (Pun not intended.) It's easy for us to see, with hindsight, what the consequences of massed firepower would eventually be. For the people at the time, it was not so, even though for them it was quite literally a matter of life and death.

Source: Geoffrey Wawro, Warfare and Society in Europe, 1792-1914 (2000) All wikipedia links only for illustration.

Piggybacking on this: Does AskHistorians think that Paddy Griffith's theories on why this happened, laid out in Forward into battle, are valid?

It's about development of tactics to adapt to technology.

In the early years of gunpowder weapons, they were relatively inaccurate, being muzzle-loading muskets usually, and the only way to reliably hit anything was to send as much ammo downrange as possible. This meant that the most reliably fighting formations was in rank and file troops, marching in formation and firing in volleys, with each soldier not aiming at any single particular enemy. Skirmishers using hit-and-run tactics might have longer muskets what were longer than a man is tall, improving accuracy to let them hit individual targets, or they might have early muzzle-loading rifles, which were accurate but were more difficult to load.

With the invention of breech-loading weapons, cartridge ammunition, and automatic and semi-automatic weapons such as the M1 Garand and the Gatling gun, tactics began to adapt to the new technology. As we learned in the American Civil War, troops get absolutely annihilated when marching rank-and-file into battle against accurate, high rate-of-fire weapons.

So, slowly but surely, tactics evolved into squad-based tactics, today called "fire and maneuver", essentially relying on maneuvering your forces to outflank the enemy, relying on supressive fire to keep the enemy from returning fire and accurate shots by individual soldiers to make a kill.

This shift in focus from mass volleys to troop mobility led to a reduction of size of the basic troop units, down to 4 or 5 man squads or fire teams.