Line infantry tactics really started to decline around the time of the American Civil War. The widespread adoption of the rifled musket meant that infantry could fire far more accurately and at a greater distance. The purpose of the close packed mass formations that characterised the wars of Napoleon was to ensure that the greatest number of muskets were able to be brought to bear on the enemy. Mass firing inaccurate muskets was the best way of maximising the number of potential casualties an infantry unit could inflict. As muskets grew more accurate, the capacity for killing at greater range also grew and that meant infantry didn't need to formed as tightly. Additionally, the increased range and accuracy also reduced the potential effectiveness of cavalry. Tightly packing infantry also provided a level of protection from cavalry which was most effective on loose and broken formations. Developments in artillery also affected the way soldiers fought. Shrapnel shells and explosive shells coupled with rifled cannons increased range and killing power especially when used against tightly packed formations. These developments led to a loosening of infantry formations and a greater level of emphasis being placed on skirmishing.

The Franco-Prussian War saw a continued use of line formations as military thinking and tactics had not caught up to the technology. The appearance of the Gatling Gun (which saw some limited service in the American Civil War) essentially signaled that the days of massed infantry formations were numbered. By the end of the 19th century, breech loaded weapons had replaced muzzle loaded rifles, greatly increasing the firing rate of the infantryman. Smokeless powder also led to the continued growth of the emphasis on skirmishing as the clouds of smoke produced by black powder weapons no longer hindered accuracy or obscured infantry formations. By the Boer War, extended order had replaced close order formations in the British Army, a tactic more suited to type of warfare that conflict required. Extended order is a looser formation of men spread over a wider area and emphasises skirmishing over massed fire.

Of course, by the start of the First World War, line formations were completely obsolete and to continue utilising them would mean disaster. The Russo-Japanese War had signaled the destructive potential of modern artillery and machine guns and although not all the lessons from that conflict were listened to, it was clear to all belligerents that extended order and loose formations would be far more effective going forward.

It's important to stress, I think, that the transition from line to skirmish tactics was not a flick of the switch. If you look at, for example, Napoleonic-era warfare, you find troops moving pretty fluidly between line and skirmish formations to suit the needs of the moment. A good illustrative example that springs to mind personally is the French right wing at the Battle of Aspern-Essling on 21-22 May 1809 (see this map for reference), which we have a particularly harrowing firsthand account of from then-Sergeant Jean-Roch Coignet of the Guard Grenadiers. (All emphases mine.)

At 11 am on 22 May, the Imperial Guard crossed over the Danube from the Lobau and marched out in the direction of Essling. To quote Coignet on the Guard's movement,

We crossed one end of the island, and came to another bridge which we went over at the charge. The foot chasseurs crossed first, dashed into the field, and wheeled to the left in column instead of to the right. This mistaken movement could not be rectified, and we had to go into battle at once, with our right wing near a branch of the Danube.

The Guard then held its position in front of a large Austrian artillery battery, deployed just outside of Essling. From Coignet:

The fifty guns of the Austrians thundered upon us without our being able to advance a step, or fire a shot. Imagine the agony we endured in such a position, for I can never describe it. We had only four of our own guns in front of us, and two in front of the chasseurs, with which to answer fifty. The balls fell among our ranks, and cut down our men three at a time; the shells knocked the bear-skin caps twenty feet in the air. As each file was cut down, I called out, "Right dress, close up the ranks!" And the brave grenadiers closed up without a frown, saying to one another as they saw the enemy making ready to fire, "The next one's for me." – "Good, I'm behind you ; that's the best place ; keep cool."

A ball struck a whole file, and knocked them all three head over heels on top of me. I fell to the ground. "Keep cool," I called out; "close up at once." – "But, sergeant, the hilt of your sabre is gone, your cartridge-pouch is half cut off." – "That's nothing; the battle is not yet over."

...The losses became very heavy. We had to place the guard all in one rank so as to keep up the line in front of the enemy. As soon as this movement had been made, a stretcher was brought up on our left, borne by grenadiers, who deposited their precious burden [the wounded Marshal Lannes] in our centre.

There is plenty more in the account's description of the Austrian cannonade, but I won't restate the whole thing in full – you can read it for yourself on archive.org, the relevant part begins on p. 174. But as you can see above, during the thick of the action against the Austrians the Grenadiers remained deployed in line, even when losses were so heavy that they had to form one-deep. Then, however, the French tried to take action against the artillery. Fleeing elements of Lannes' corps, who had retreated behind the single-line Guard for cover, were rallied by Marshal Bessières:

The village of Essling was in our possession, though it had been taken and retaken and burnt. The brave fusiliers remained masters for the rest of the day. The soldiers behind our file being somewhat restored to presence of mind, Marshal Bessieres came up to them, and reassured them by saying, "I am going to take you forward as sharp-shooters, and I shall be among you on foot myself."

Then they all started off with this brave general. He then placed them in extended order within range of the fifty guns whose fire we had stood since eleven o'clock in the morning; and thus there was a line of sharp-shooters to protect the file-firing which had been opened on the Austrian artillery. The brave marshal, with his hands behind his back, walked up and down the line, silencing for the moment their fury against us. This gave us a little breathing-space; but time passes slowly when one is awaiting death without the power to defend one's self. The hours seem ages. After having lost a fourth of our veterans without having burnt a priming, I was no longer at a loss for sergeant's stripes and epaulets; my grenadiers brought me my pockets full. This terrible battle cost us dear. The brave marshal remained behind his sharp-shooters more than four hours... At nine o'clock the firing ceased.

So what we can see here is that when it came time to try and silence the guns, the infantry of Lannes' corps moved out in loose formation, with the skirmisher screen and line behind it working in tandem to try and lay down a sort of suppressive fire on the Austrian guns.

The Aspern-Essling case highlights two key features of how line and skirmish order worked in Napoleonic tactics: Firstly, the two operated in tandem rather than mutually exclusively, as a formed body of infantry behind would, ideally, be screened in front by troops in extended order. Secondly, the same troops could be called upon to perform both roles: Lannes' corps had fought not just in line in the open but indeed in more chaotic non-formation in the urban conditions of Essling, and then was transitioned to skirmish screen for the Guard when needed.

Just to highlight the fluidity aspect a bit further, one of the most noted light infantry formations of the Napoleonic period was of course the 95th Regiment of Foot (Rifles) in the British army, a force of rifle-armed troops that were distributed as the army's specialist skirmisher force, though also functioning alongside faster-firing, musket-armed light infantry contingents. During most of the Napoleonic Wars, particularly in the Peninsular War of 1808-14, the Rifles fought as skirmishers, but there were exceptions. Notably, at the Battle of Waterloo in 1815 the Rifles actually fought in line at several points. Lieutenant Colonel William Eeles of the 95th, a captain in the 3rd Battalion (part of the 3rd Brigade) at the battle, had this to say in a letter recalling his experiences in 1834:

The Brigade continued to advance through the smoke until it passed beyond the crest of the British position; on the smoke clearing away, the 71st Regiment, with whom my Company of the 3rd Battalion 95th Regiment was then acting , found itself while in column very close to and in front of a large body of the Enemy's Infantry, formed in line, and dressed in grey great coats The 71st immediately formed line, and I placed my Company of Rifles on the right of that Regiment. I can only here observe that the French and 71st were closer than I ever before saw any regular formed adverse bodies, and much nearer than troops usually engage. The French opened a very heavy fire on the 71st, who, nevertheless, completed their formation in the most regular and gallant style. I formed my Company on their right, and in line. During this operation the 71st and the Company of the 95th suffered severely, but immediately on being formed succeeded in repulsing the Enemy, who retired almost unobserved in the smoke.

Subsequently, the 71st and Eeles' company came under attack from some French cavalry, and

We had just time to get back and form in rear of the 71st Square, when the Enemy attacked that Regiment with much impetuosity and determination.

In fact, Eeles' company was not alone in forming up in dense order that day. In his own account he notes that during the French cavalry attack, the 2nd Battalion of the 95th, which had remained as a single battalion rather than being divided into its constituent companies, formed a square all to itself. After the cavalry attack ended, the 3rd Battalion reformed, and did as follows:

The 3rd Battalion 95th Regiment... as the 52nd and 71st opened out a little, was formed in line between these two Regiments . When the smoke cleared away a little I found that we were moving between both Armies, and driving away some French before us in the greatest disorder. I was almost immediately ordered out to skirmish with my Company, and continued advancing in that manner until some English and German Dragoons, followed by some French, passed along the front of the Brigade.

Eeles' company again returned to line, and then, after encountering and driving back some regiments of the Old Guard, again spread out in extended order to pursue.

The summary I applied to Aspern-Essling above similarly works for Eeles' company at Waterloo. There were times when line was more sensible, and times when skirmish order was. There was a significant period in which skirmish and line formations not only coexisted but also complemented each other. While the eventual trend would favour open order, we shouldn't think of this as an instant inversion, but rather the gradual winning out of one of two simultaneous modes of infantry tactics.

It is just about impossible to draw a common line across various 1st rate powers' tactical and doctrinal adjustments to more sophisticated technological and organizational developments with any sort of precision. In other words: there is no simple answer of a single battle or war. The French Army during the Napoleonic Era - for example - saw what can be described as a relative loss of tactical sophistication over the course of those wars.

Prior to 1805 the Grande Armee was reorganized into corps, centralized, and vigorously trained during the peace of Amiens and prior to the War of the 3rd Coalition. L'Ordre Mixte - mixed order - of "line" forces was widely adopted: this meant loose order skirmishers out front, followed by ranked lines in the center and marching columns on the flanks. When supported by guns and cavalry, this flexible formation allowed for quick action against a variety of enemy dispositions (and the early opponents of the French used much more rigid linear battalion formations - quite similar to those of the British Regulars fighting in North America in the late 18th century, and which played right into the hands of tactically adroit French officers). This tactical formation was employed with some variation up to the divisional level, and combined with fast moving independent corps, and bold strategy by the Emperor, the French Army became a lethal instrument.

Later in the war (by 1809 in particular), a combination of veteran attrition, the increased use of less experienced conscripts and incorporation into the French Army of non-French allied formations, the expansion of the war into more theaters, and the growing sophistication in organization and tactics employed by their principal enemies necessitated a devolution to using more column tactics by the French Army. This made for unsophisticated formations meant for punching a hole through enemy lines by relying on the weight of numbers (still the most decisive of factors in battles at the tactical level between similarly equipped foes, and the real change between 1815 and 1914), but sacrificed flexibility, maneuverability and sustained huge losses when exposed to enemy fire due to their densely packed ranks. Wellington said of these tactics following Waterloo:

Never did I see such a pounding match. Both were what the boxers call gluttons. Napoleon did not maneuver at all. He just moved forward in the old style, in columns, and was beaten off in the old style.

I used the example of the French to illustrate the constant tactical fluctuation and organizational style adopted in order to adapt to changing war conditions. The other part of the question has to do with the increased dominance of firepower on the battlefield. As I stated before, prior to 1914 the number of troops brought against the point of attack was still the most decisive factor in a pitched battle (obviously field problems like river crossings, defiles, fortifications, etc. as well as the skill of leadership can change this calculation significantly). Similarly armed men led by competent officers generally fight until one sides' morale breaks - this tends to coincide with the arrival of enemy reserves when yours aren't to be found, or if one side maintains the advantage in numbers at the point of attack (again, outside factors obviously affect this, and there are exceptions - which stand out because of their relative infrequency).

The change - which took place between the start of the Crimean War and the end of the Russo-Japanese War - was that armies now brought enough sustained, accurate and deadly firepower to negate huge numerical superiority on the part of the attacker. Numbers could no longer 'guarantee' success in ways they had since the dawn of organized warfare, and the firepower a prepared defense could bring to bear against the numbers required to overcome its resistance was far out of proportion to anything experienced before. The resulting changes undertaken to tactical organization and the methodology of command and control initially fell somewhere between redundant and irrelevant, which is unsurprising given the notorious institutional inertia of established military thinking and practice. By 1915 the Great Powers had fallen into a forced mutual 'siege' of opposing fortifications where the armies stayed in close contact with one another constantly, and attrition was the main byproduct of a never ending quest to finally achieve enough local superiority to overcome enemy resistance, which neither side could accomplish due to the rates of attrition. It was the catch-22 that lay at the end of organized warfare when brought to its logical conclusion.