In 1441, the Papal State ceded the town of Sansepolcro to the Republic of Florence. Diplomats from the two states met and stipulated a formal agreement in which the town passed to the Florentines and the new border was placed on the edge of a stream called 'Rio', which in Italian simply means small river. Too bad there were two streams of the same name in the same area. The Florentines therefore considered the stream further north, towards Florence, as the boundary, and the Pontiffs the one further south, towards Rome. In the small strip in between, just over 500 metres wide, was the village of Cospaia, which suddenly found itself independent of both and proclaimed itself a republic, with the agreement of the two powerful neighbours who preferred that solution to a new treaty. The little republic would remain independent for the next 400 years. This funny little story illustrates how, in already densely populated and known territories, states based their borders on precise geographical references (or thought they were) and maps were drawn accordingly, not the other way around.

But it sometimes happened that diplomats only had to use maps because the territories were very distant and poorly known. For example, the extreme western border between the United States and Canada. The Oregon Treaty of 1846 said that 'along the forty-ninth parallel of north latitude to the middle of the channel which separates the continent from Vancouver Island, and thence southerly through the middle of the said channel, and of the Strait of Juan de Fuca, to the Pacific Ocean'. Too bad the real channel had more islands than the inaccurate maps available showed, and so there were two navigable channels separating the continent from Vancouver Island, leaving it uncertain which was the real border and to whom the islands in between belonged. The uncertainty dragged on for almost thirty years and saw the dispatch of troops from both contenders to reclaim the middle islands, although the only casualty of the tension was a fat pig and some potatoes.

So, to answer your broad question.... it depends! When you had very precise geographical references, you used those directly to establish a border: mainly large and small rivers, but also mountain ridges, hills, villages. And then maps were made. When, on the other hand, the borders of distant lands had to be established, first maps were made and based on those, it was decided where the border would run. In both cases mistakes were possible, which were settled either by new agreements or even became grounds for wars.

Translated with www.DeepL.com/Translator (free version)

The great majority of borders in history result from wars; conquests by show of force or purchase; or sometimes peaceful secessions, and thus representatives of the opposing nations would come to some agreement in a treaty negotiation. Disputes and unresolved ambiguities about the details of those treaties occasionally led to further warfare. Some nations have boundaries along mountain ranges or water features whose origins are so ancient we know little about how they came to be, and some of these boundaries remained uncertain, and unmarked, until quite recently. The principalities divided by the Himalayas or the Pyrenees come to mind, or tribal lands and emirates on the Arabian Peninsula that were separated by empty desert.

The practice, starting around the 18th century, of dividing up unexplored lands using lines of latitude and longitude, or direct lines from one geographic feature to another, led to a lot of ambiguities when various features referenced in a royal land grant or legislative act turned out to be in a very different place than had been assumed. State boundaries for the USA are set by the act of Congress that sets up a new territory, or admits a new state to the union. Then surveyors, typically in the employ of the states concerned, go out and set monuments that become the accepted, on-the-ground boundary line. The most famous of these surveying parties were Mason and Dixon, who in the 1760s surveyed the line between Pennsylvania and Maryland that bears their name. Other contractors were not as competent: The Clark Survey of the Texas-New Mexico boundary in 1859 set it more than two miles too far west. This article in American Surveyor magazine quotes a congressional report describing it as “Perhaps the Most Incorrect of Any Land Line in the United States.”

Some state boundaries were adjusted for convenience and to benefit trade. As Congress began to divide the Northwest Territory into states, the intention was to set borders at the southern tip of Lake Michigan. That's neat on a map, but ignores the reality of 19th century transportation, which was dominated by lake and river navigation. The northern border of Ohio was adjusted so that the mouth of a key river, the Maumee, would be within the new state. That caused friction with Michigan, resulting in the Toledo War.

A few years later, Indiana's northern border was also adjusted 10 miles northward to give it access to Lake Michigan, which since the opening of the Erie Canal in 1825 had become economically very important. In a similar vein, the boundary of Illinois was pushed northward nearly 60 miles so that the Chicago River, one end of a planned canal connecting the Great Lakes and the Mississippi, would be within a single state.

I would assume there are similar tales of mistake and ambiguity leading to dispute and warfare to be found among the suspiciously straight lines colonial powers drew on the continent of Africa, and in the Middle East, but that's not an area I've studied.

The details of state lines are given in Boundaries of the United States and the several states, a US Geological Survey book available as a PDF here or older versions on archive.org or Google Books. Mark Stein's book How the States Got Their Shapes is a popular book telling the same tales. Stein also did a follow-up book, How the States Got Their Shapes Too: The People Behind the Borderlines, that tells the human stories of some of the land disputes.

Bill Hubbard Jr.’s 2008 book American Boundaries: The Nation, the States, the Rectangular Survey also relates the history, combined with a superb history of the Public Land Survey System and how it was deployed across the West.