Specifically between the year 1700 to the 1710's - 30's, particularly the British, French and Spanish navies.
When I think of a galleon I imagine a 17th century-styled 1 or 2 gun decked, 2 - 3 masted ship with an exaggeratedly curved bow and stern, but I'm not sure if the term galleon was still in use after the 1600's. When I imagine a man o' war or ship of the line, I attribute them with huge ships of 3 gun decks and 3 masts that are seen in late 18th century and Napoleonic-era paintings. I remember from reading British Warships in the Age of Sail 1603 - 1714, I looked at the section of the book of anything between the years 1705 - 14 and there were illustrations of warships with 3 decks labelled as ships of the line (or man o' wars, I forget which name), but also warships with only 2 gundecks labelled with the same names, so it got me confused since I have seen illustrations of 2 gun decked warships labelled as frigates, and I usually associate frigates of the time period as ships with 3 masts and 2 gundecks.
Were the terms "ship of the line" and "man o war" used loosely between kinds of warships in the early 1700's? Or were they used in a specific way, judging based on the number of cannons a ship carried and the number of masts it has?
Hi!
So firstly I think what we need to work out is some exact definitions for the words you're using. A "Man 'o' War" was not a specific ship, but rather the general terminology used by the Royal Navy for a "Kings Ship", or a naval battle ship. This could refer to a sloop, frigate, or ship of the line, but was used to distinguish between a merchantman and a battle ship. Hence the slang for sailors on battle ships (man-o-wars-men). I believe that the etymology of the term is actually the other way around though, as battleships were referred to early on as "men of wars ships", as in ships that carried men of war (soldiers), and eventually the ships was lost and the nickname man-o-war remained.
A 'Galleon', as you have rightly pointed out, was a multi-decked armed ship, though not always with exaggerated bows and sterns. Originally this referred to armed merchant ships, hence the stereotype of the 'Spanish treasure Galleon', which was mainly a cargo ship that was heavily armed to protect itself.
A Ship-of-the-Line is a reimagining of the phrase 'Line of Battle Ship', and referred to the ships that would fight in the line of battle. These would be the biggest and most powerful ships that the navy could field.
In regard to your confusion, a ship of the line could have been a very different size depending on what era you were looking at, obviously the Royal Navy's ships in the Napoleonic era were much bigger than those in the 1600's.
So in answer to your first question, the terms "Ship of the Line" and "Man o War" WERE in fact used loosely between various warships.
For specific ships, the Royal Navy had a rating system for their ships. For example the famous HMS Victory was a 'First Rate Ship of the Line', of 104 guns and three decks. A second rate would have around 90-ish guns, a third rate 64-80 etc.
Ships from 1st-4th rate would be referred to as 'Ships of the Line', whereas a 5-6th rate ship would be a Frigate. This is why you would be thinking of a Frigate as being a 2 decked ship, although 2 decks would be a maximum, they would much more likely be one deck.
The number of masts and rigging style affected the classification of the vessel. To be classified as a ship the vessel had to have 3 masts, all of them square-rigged. Otherwise they would be referred to as Schooners, Brigs, Sloops, Cutters etc
Sources:
Uden, Grant; A dictionary of British ships and seamen (1980) Dull, Jonathan; The age of the ship of the line : the British & French navies, 1650-1815 (2009)