Why not spread out, flank, break formation and use the natural environment for cover to skirmish/employ some Guerilla tactics?

Many units did this. They were called skirmishers/light infantry. Skirmish/Open order had men in either small groups or pairs fighting before the full battalions came into contact. They were supposed to harass the incoming battalions, but ended up fighting with the enemy's skirmishers they were trying to do the same thing. Skirmishers would often fight in pairs, one would fire while the other would cover him while he reloaded. They would always have one loaded weapon to defend themselves. Their job was to pick off enemy officers and NCOs which would cause chaos among the now leaderless men. Light infantry was also the preferred force for fighting in towns and buildings. While defending buildings they would loophole walls and knock out windows to fire out of at advancing enemies, something that modern soldiers still do.

Another light infantry tactic was called chain order. Ahead of a reserve force, files of 4 men would be spaced about 10 yards apart. The men in the file would stand in a line shoulder to shoulder facing the enemy's advance. The man of the right would advance a few paces and fire. He would then fall back to the left of the chain to reload. Then next man would fire, fall back, and reload etc. The advantage of the chain over a pair was that the chain was able to keep a steady rate of fire while a pair would inevitably have to wait for the partner to reload.

So why not have all the infantry fight in this manner?

The simplest answer is that smooth-bore muskets are not very accurate. Soldiers did not get much, if any, practice actually firing their weapons. They would be drilled in firing and reloading, but regular line regiments did not spend money for target practice. Elite and Light regiments were trained in skirmishing and would be able to practice, but not as much as you'd think. Elite British Riflemen were allowed only 60 rounds of practice for a year. British line infantry were allowed 30, whereas Austrians regulars were allowed between 6 and 10. In reality, new conscripts might get a few weeks of training and then only fire their weapon once or twice a year for practice. In open order your average infantry man wouldn't be very dangerous. Even in line you're looking at much less than 50% accuracy. Prussians did a test in 1755 of two companies of Grenadiers(the elite companies of a battalion). At 300 paces they fired at a target 10 paces wide and 10 feet high, their accuracy was 12%. At 150 paces it was 46%. Massing together 500-1000 men in a line was the only way to get satisfactory result. Also, since they did not have a lot of training being together in large units gave them some confidence.

Desertion was a problem even before a battle, so letting squads of men roam about the battlefield wouldn't be smart; they'd do their best to stay out of harms way or flee completely. The line provided a structure for the officers to control the rankers. At each corner of a company deployed in line was an NCO. The NCOs were in charge of keeping the line in order. The captain stood to the right or left and of the company directing it. Directly behind the line stood the company's lieutenant(s) with sword drawn. One of their duties was to make sure no one fled backwards.

Next, the reason why all infantry was not in open order was cavalry. The only defense infantry men had against cavalry was forming a square. The men would be shoulder to shoulder in lines 4 deep with the first rank pointing their bayoneted muskets out. Horses would veer off instead of running headlong into hundreds of bayonets. All the while the men of battalion would be firing into the horsemen. It was very rare for a square to be broken by cavalry and was generally due to panic infantry running rather than the cavalry themselves.

Dictionary of the Napoleonic Wars - David Chandler

Tactics and the Experience of Battle in the Age of Napoleon - Muir

With Musket, Cannon And Sword: Battle Tactics Of Napoleon And His Enemies - Nosworthy