Well, by the time of the 18th century, the formalism of the line of battle was falling out of fashion, as it was seen that line-of-battle, uh, battles, often didn't produce decisive results. I wrote more about this here, here and also here if you'd like to read about that.

In terms of smaller ships, though, you're correct in thinking that they did not form in traditional lines of battle very often (partly because it would be fairly unusual for several frigates or brigs or other ships to operate together, they were too valuable as scouts and messengers to be very often sent out in groups). If we take a hypothetical encounter between two frigates, for example, the pre-battle maneuvering would be entirely taken up by trying to gain the weather gage (that is, an upwind position) that would allow the ship with the weather gage to control the engagement. Depending on the nature of the captain, ships might engage in maneuvering at a distance or might try to simply close and fight it out, gun to gun. If you're interested in ship-to-ship engagements of the period, there are a few good, easily accessible books out there:

• Toll, Ian W. Six Frigates, Norton 2006

• Miller, Nathan. Broadsides, Wiley 2001

• O'Brian, Patrick. Men of War: Life in Nelson's Navy, Norton 1995