This question is understandably asked relatively frequently here, and is in the FAQs. http://www.reddit.com/r/AskHistorians/wiki/war#wiki_line_formations_aka_linear.2Fnapoleonic_tactics

Consider a few things:

1. They didn't have radios on battlefields that were covered in a thick smog of smoke (way before the guns of today which produce little to zero smoke; they produced billows of them every shot) and were incredibly loud (muskets were much louder than modern rifles)

2. They were using wildly inaccurate weapons

3. Being with no armor and highly inaccurate, one shot at a time boomsticks they were extraordinarily vulnerable to cavalry individually

4. Due to the inherent lack of individual firepower large masses of 'bad guys' could swarm smaller groups with relative ease and little casualties

5. Armies were, generally, comprised of people who didn't want to be there. As we move into the 18th century armies become less personal tools of mercenary bands temporarily raised and dropped and more standing forces; an arm of the nobility to have personal growth and power and for the monarch to exercise his control. Standing armies require standing men and, in the general, these were poorly paid peasants or forcibly impressed prisoners or flat out kidnapped people. Desertion was high for a reason.

Now you need to combine all of this and you get a pretty clear picture why linear tactics were used. The battlefield was far more chaotic and without modern communications...men had to be kept close together for their officer to relay orders and to keep them in check. The men were, in general, uneducated peasantry or people who generally didn't want to be there; blind, mindless drill of the line formation was far more effective as it simultaneously allowed officers to make sure no one got any funny ideas of running away while also making it so simple an idiot could do it. This has the added bonus of building camaraderie; you got men to your side and your backs and you all are in it together and you aren't going to abandon them.

Then we get to the more practical, military theory reasons for it all. Their weapons were inaccurate but incredibly loud and make a shocking amount of smoke. Psychological warfare is used today and it was used back then and linear tactics are, fundamentally, psychological warfare. It's not that scary (relatively) having a few hundred muskets spread over a couple hundred meters kind of going off sporadically and 1-2 bullets hitting their mark periodically; it is significantly much more horrifying to have hundreds of muskets going off at precisely the same time and hundreds of bullets hitting their mark at precisely the same moment while your eardrums shatter and smoke clouds billow toward you. Even if you don't get hit you see 4 people on either side of you drop and then a wave of bayonets rushing right for you...you will be running then.

Consider also more professional armies such as the Habsburgs or especially the British; they were capable of doing things like 'firing by rank' where the first row would fire, kneel and reload while the second row fired, kneeled and reloaded and the third row fired and kneeled to reload and then the first row fired again. Bam bam bam bam bam; almost semi-automatic fire of entire rows letting loose fire; that was only possible with tight knit formations that could coordinate like such and it was a psychological horror.

And then we get to bayonets. It's much more appropriate, as I toot on endlessly, to when first getting into this era of military history to think of pikes being replaced with bayonets that can shoot. It really gives the idea I'm trying to get across here; the bayonet was the principle decisive actor not the musket part. The musket part was the prodding and "initiating" phase ("in general") and the bayonet end was the actually winning the battle phase. The bayonet, when combined with 150 men standing in tight formation, allowed them to not only stand up to but decisively ward off cavalry formations whereas hundreds of men spread out are highly susceptible to armored men on armored horses carrying giant fucking lances and sabres chopping you all down. In other words the common phrase united we stand, divided we fall...to cavalry.

As both musket technology improved and more national armies became 'a thing' (after the French Revolution particularly), i.e. armies of people forced to be there replaced by people who more or less wanted to be there through national pride or whatever else, low level skirmisher operations were fully reasonable. Professional light infantry detachments would develop throughout the Napoleonic Wars though; the issue just comes with the fact that they require highly trained, highly experienced, highly motivated troops to perform skirmishing activities (both out of battle raiding and scouting and in battle screening); that came pre-French Revolution from such like the Austrian Grenzers ("soldier peasants" who operated in an almost DMZ cordon against the Ottoman border) and the Jager and eventually the French Voltiguer and so on. You can read more on this evolution in my post here.

TL;DR:

1. Yes there were negatives to being in the open, notably being more susceptible to enemy fire, but battles were (in general) won and lost on the backs of bayonets and cavalry and not "shooting at each other contests" and with that it makes sense why they would stand together.

2. Even in shooting it out it was preferable; yes they were more susceptible but they also exponentially increased their volume of fire and psychological impact of fire.