SO before really getting into the meat of this answer we should probably dispell some misconceptions you have. First, this is more normally referred to as the Napoleonic style tactics. He didn't invent them but he did to a degree perfect them. There is no typical "colonial warfare" but in most instances on the North American continent conflict between Natives and Europeans took place on a small scale and utilized raids and ambushes, meeting engagements/ set piece battles were incredibly rare.

So why did armies fight in close order like this? I would argue that there were three primary reasons this mode of warfare came to be, command and control (C2), massing of firepower with inaccurate, short range weaponry, and unit cohesion.

The need for command and control is often times forgotten when discussing battlefield tactics by those unfamiliar with them. The level of organisation required to get 10,000 to 100,000 men to all act in concert is incredibly difficult today with advanced telecommunication equipment including, satelites, radios, and Blue Force Trackers. Imagine how difficult this must have been in ages past prior to the advent of even the telegraph. All orders must be given either through audio or visual signals made by an individual with his hands or his voice. If everyone is grouped together in tight formations this tasks becomes much easier. Especially if everyone is wearing bright colored uniforms that stand out against the natural vegetation.

Say you have a company of about 100 men armed with muzzle loading muskets. you can either spread them out group them together in close order. Say you do decide to spread them out in a line with 1 meter between each man, your formation is now 100 meters long and there is no way you can communicate with your whole company at once and if you are in dense vegetation there is no way you can even see them all either. You can see how this can break down very quickly, and you are only one company out of hundreds in your army. Yes there were some outfits that fought in a more modern fashion including Roger's Rangers and German Jaegers but they were but they were not used in open battle and rarely determined the victor.

Now let's consider massing of effects. In any battle you want to mass your effects as much as possible. Lets take that 100 man company of infantry again all armed with muzzle loading muskets. First of all they aren't going to be effective beyond 100 meters, secondly they are at peak efficiency only going to fire three rounds a minute. If your men are spread out and fighting they are simply not going to be able to mass their effects enough to break an opposing unit whether they are fighting in close order or spread out like you. If they are in close order you simply aren't going to be able to inflict enough casualties to make them retire from the field and if they so choose they will easily break your lines (which I'll get to in the next paragraph). Now lets say you close ranks and fire massed volleys at the enemy. Your fires are now much more concentrated and if you fire in volleys you are sending 100 rounds at the enemy at once. This is going to have ea much greater effect on the enemy than one or two rounds flying at them at once. After firing though its not a good idea to lie down and reload considering your weapon will likely be almost as tall as you are and it needs to be upright so your powder and ball don't just fall out.

Lastly you have to consider unit cohesion. A group of 100 men standing in close order is going to have a lot more never than a scattered band of 100 men in most situations. If a man is isolated and has no one behind him he is a lot more likely to flee the enemy, especially if that enemy fixes bayonets and charges straight at him. A cohesive unit is much more likely to stand and fight than one thinly spread over the area of a football field.

It basically boils down to the the fact that generals had to be able to control their formations which is made much easier if everyone is arranged in neat rectangles, they need to mass their effects on the enemy in order to have an effect on him, and because if the two sides meet in a melee you want to make sure every man stands and fights and doesn't just flee because there isn't anyone around him to help him out.

I hope that answers your question.

These tactics that you describe have their origin in the 15th Century, however, based on the wording of your question, I will ground my answer in the Colonial Period time frame of the 1620s-1770s. During this time period, arquebus underwent great changes in their mechanics and tactical usage, eventually evolving into muskets that you are probably more familiar with. Warfare was actually quite different at the beginning of the Colonial period in America than at its close.

Spanish armies in the early 1500s typically used infantry formations called Terichos or "Pike and Shot." These formations contained pikemen and arquebusiers (about 60% to 40% make up) in a type of "bastioned square" formation. The strength of the Terichos is that they largely negated the role of cavalry in battle, as horses will not run into a solid block of men with pikes. The arquebusiers provided additional firepower to harass enemy formations, but by and large, pikemen did the majority of the work (Source: Arfaioli, Maurizio. The Black Bands of Giovanni: Infantry and Diplomacy During the Italian Wars (1526–1528). Pisa: Pisa University Press, Edizioni Plus, 2005. ISBN 88-8492-231-3.)

By the English Civil War (1642-1651), arquebus had evolved into the matchlock musket, a much more reliable and lighter firearm that could be fired quicker than its predecessor. In addition to this, Gustavus Adolphus of Sweden had revolutionized armies during the Thirty Years War as his usage of muskets and artillery helped neutralize the pike and shot formations. Armies still contained pikemen, but they became increasingly ineffective as a rapid fire from lines of musketmen and artillery could decimate their opposing pikemen filled ranks. Pikemen were still necessary though, because attaching a bayonet to the barrel of a matchlock musket took away its ability to fire. With more and more emphasis being placed on musketmen, cavalry became increasingly relevant again (Brzezinski, Richard. The Army of Gustavus Adolphus. Osprey Publishing (1993). ISBN 1-85532-350-8.).

By the time of the French and Indian war (1754–1763), muskets had evolved to the Flintlock version you are familiar with, best illustrated by the British Brown Bess. Flinklock muskets contained bayonet lugs, thus negating the need for pikemen, as a musketeer could "fix bayonets" and charge, while maintaining the ability to still fire rounds. This way, musketmen could fill the role of musketeer and pikemen, by switching formations from line-fire to square in case of cavalry charges. This last point is critical, because if you broke up formations, then cavalry could just demolish a scattered infantry force.

Now, with all that background out of the way, was linear formations the best? Yes and no. Despite all these technical and tactical advancements, muskets were still inaccurate and only effective at 100 yards. In addition, though they load and fired quicker than their predecessors, a skilled soldier could only fire about thee to four rounds a minute. So, because of the weaponry, infantry formations needed to produce a high rate of fire in mass volleys for great effect. However, during the Napoleonic Wars, Napoleon was famous for using a column formation. These columns were deep rectangular shaped formations that would fire at a shower rate than typical straight lined infantry four deep. Despite this slower rate of fire, once columns made hand-to-hand contact, the depth of these formations quickly overwhelmed line defenses and broke many a Austrian and Russian army (Arnold, James R. "A Reappraisal of Column Versus Line in the Napoleonic Wars" Journal of the Society for Army Historical Research LX no. 244 (Winter 1982): pp. 196-208.)

So, yes, stand in line and shoot formations were absolutely the best tactic, because of the inaccuracy of the weapons and the short distance.

There are some good answers in this thread, but I wanted to add some stuff:

There were a lot of folks who did not fight in the classic linear formation during this period. I am mainly familiar with the 18th century, and as the century went on, European nations increasingly experimented with what they called "Light infantry." As compared to regular infantry, light infantry acted as skirmishers, ranging out ahead of the regular infantry in loose formation, attacking officers and so on.

The light infantry had various different weapons. Often, they used normal muskets, which weren't very good for aiming, but I suppose even trying is better than nothing. Later on, there was an Austrian attempt at an air rifle and, of course, rifled weapons became more common (Though, never the norm). The French Revolutionary army was the most successful user of light infantry and one of their main tactics in the early conflicts of the French Republic was to employ tons and tons of light infantrymen, who would continually wear down the dense formations of their enemies and be difficult to hit with a counterattack.

Anyways, tactics during this period were a lot more complex than you've implied. If you want to read more, I'd recommend Fighting Techniques of the Napoleonic Age. It's a decent book. Also, if you are interested in fiction, I recommend Rifleman Dodd by C.S. Forrester. It is a fictional story about a British light infantryman fighting the French during the Peninsular War. Quite good, really.

Here are several prior threads on the subject. 1, 2, 3.

I've always wondered this - In Napoleonic style tactics, why were bow & arrow not used, or at least tried? It seems they could be just as accurate (if not more) than a musket at a similar distance, with a fire rate +- 2-3X (considering maybe a rate of around 3-4 rounds per minute with a musket, and a rate of around 8-10 rounds per minute with a bow). Since armor went out of style post firearms with their ability to penetrate (I assume, please correct if there was another reason for the decline in armor use), would this have given the bow-wielding army an advantage? Or was a musket ball considerably more devastating than an arrow, so as to take the enemy immediately out of the fight?

I apologize in advance for the assumptions, I estimated the fire rates per minute based on meager muzzle loading and archery experience; I've just always been curious about this.

I remember reading a work on the Battle of Malplaquet (and the War of Spanish Succession) in general and the part which struck me was a first-hand account of a battalion of the line (ie the generic foot soliers) going up against a similar unit. First, the battle had been raging for a good part of the day and the smoke was thick enough that visibility was incredibly limited. The engagement took place at close range, the men were aiming at muzzle flashes rather then clear targets, and they relied on crushing platoon fire to drive the enemy off (platoon fire being designated sections of the unit all firing at the same time, and them being staggered so by the time the last section has fired the first is reloaded).

The passage also highlighted how much, as prior posters stated, command and control was critical. Battlefields are incredibly loud places and keeping the unit cohesion and executing these fires required both discipline and the men being densely packed. Plus, morale wise, being in tight formation actually helps increase your morale. You feel more confident and capable.

Finally while it was the norm in many European battle, the strict line formations were not the norm. Mathew Spring argues in his book "With Zeal and Bayonets: the British Army on Campaign in North America 1775-83" that they didn't actually use the stereotypical formations we see in the media (for example in The Patriot). Instead they fought in much looser formation and relied heavily on moving rapidly toward the enemy, counting on high morale, determination, and psychological intimidation to win the day (British units often sounded hunting calls and made a ruckus while closing to engage American units in the hope of breaking them). So there's that.