"They had limitations for sure. Surveyors' instruments were available when George Washington did a survey in Virginia. Americus Vespucci became famous for his map of North and South America. (Hence the name America.) He did it with his ship hugging the coastline as he marked out distances and directional angles of the coastline. In the 17th and 18th centuries it was fairly simple to use a sextant to obtain a fairly good fix on one's latitude anywhere in the world at any time. The sextant measures the angle of the sun or a known star with the horizon. This allowed the navigator to calculate the latitude, or the angle north or south of the equator. The problem, however, was determining longitude (east-west). This was such a wild guess, for instance, that Columbus thought he had found the "Indies"--the islands of India--rather than the Caribbean.
The solution occurred several years after a prize equivalent to about a million dollars was offered to anyone who could come up with the way to find longitude anywhere in the world. The key was in the making of an accurate clock that could not be disturbed by the rolling motion of a ship at sea. A British clockmaker solved the problem with a small, intricate clock in 1759.
Here's how it works. The earth rotates from west to east every 24 hours. The angular measure is 360 degrees for one rotation (one day). So longitude for one hour's rotation is 360 degrees divided by 24 hours per day = 15 degrees. Now Greenwich, England was marked as the line of zero degrees longitude. If the ship is five hours west of Greenwich, say at high noon on a given day, then the ship lies 5 x 15 = 75 degrees west ( or 5 hours west longitude). If the ship's navigator kept the clock on Greenwich time they could always measure the longitude. So mapmaking improved after 1759."
The usual answer to this surprisingly common inquiry is to explain that latitude can easily be determined on a clear day by checking the position of the sun above the horizon at noon, or the moon's relation to a star. And longitude can be determined simply using a precise clock, or some other, more complex methods.
That's how we got accurate coastlines of Europe and parts of Africa by the mid 17th century. But when talking about an island, something 200km wide or less, those instruments just weren't accurate enough for a detailed coastline. On the other hand, you can sail around an island, giving you perspective needed to pretty accurately sketch the coastline and locate some prominent mountains. What's much more difficult is mapping the interior, where heavy vegetation or hills may keep you from seeing very much at any one time.
Similar previous inquiries: http://www.reddit.com/r/explainlikeimfive/comments/1lnv1i/eli5_how_could_we_have_had_so_accurate_worldmaps/