Short answer: Pirates.

Longer answer: Spain just didn't have the money to protect all of its colonies so they built the big cities inland where possible.

The oldest Spanish cities in the Americas, like Santo Domingo, Veracruz, and San Juan, are all traditional ports with good natural harbors. It just makes sense to build a city on the ocean if you're a seafaring power, whose primary worry is hostile Indians, and you have to potentially supply the city by water. This is what other European powers did well into the 1700s - think of colonial cities like Boston (English), New Orleans (French), or New York (Dutch) all of which have superb natural harbors.

But, as a famous poet once said, "mo' money, mo' problems." Spanish successes in conquering the Americas made them a target. Even 50 years after Columbus, it was clear that the Spanish just didn't have enough of a navy or enough money to protect the empire from corsairs sponsored by competing European countries. For example, French pirate Jean-François Roberval hit Cartagena (now Colombia) in 1544, Sir Francis Drake hit Nombre de Dios (Panama) in 1572, and the Dutch pirate Bowdoin Henrick hit San Juan (Puerto Rico) in 1625. Fortifications are incredibly, hideously expensive to build, and that goes double when your city is very, very far from the metropole.

So Spain began passing urban planning laws under the reign of Charles I called the Leyes de Indias, or Laws of the Indies, which regulated how new settlements were to be built, and to keep the vast Spanish empire defensible against foreign corsairs. The Leyes de Indias are a milestone in urban planning, requiring things like a street grid built at an oblique angle to north for even light exposure, and a water source. But most critically for our purposes here, the Leyes de Indias required settlements to be built away from the coast if possible. That's how you get capital cities like Caracas built inland.

Did these cities still have ports? Absolutely. Caracas has La Guaira, for instance. But it's much easier to fortify and protect a small outpost like La Guaira on the sea than it is to fortify a center of government like Caracas - and it's significantly harder and less desirable for corsairs to mount a surprise attack on a capital city that's at an elevation of 3000 ft/900 m and requires you to march 15 miles/24 km inland.

These rules applied well into the dying days of the Spanish Empire, even after the golden age of piracy ended. It applied even to total backwaters like Los Angeles, which was founded in 1781. If you look at downtown L.A., it follows all of the rules of the Laws of the Indies. Downtown LA is a day's horseback ride from the ocean, the old city's streets are set on a rigid grid at an oblique angle to north, and it's built on the L.A. River, which provides a reliable water source.

/u/fiftythreestudio explains why some of the later capitals ended up in mountains, but not the early ones.

Mexico City, Bogota, Antigua, Quito, etc. are all in the mountains because that is where the largest precolonial populations were. Spanish expansion targeted areas of dense human population. More people meant more laborers and more taxes. Where possible they established their capital cities on there largest indigenous cities. Mexico City is the classic example of this.

But they also had to think about interconnecting the empire. If they followed the population, Cuzco should have been the capital of the region. It was the most important highland town as it had been the capital of the Inca Empire, but they needed to keep that region connected to the rest of the empire so Lima was chosen because it was the port (technically the port was Callao) that connected up to Panama and then back across the Atlantic.

This tension between population and connection lasted the whole colonial period. In Ecuador, project after project tried to find a better port for the highland capital of Quito, Guayaquil was too far. But no good options could be found.

Piracy only became a major problem after 1550, so any towns established before that were either fortified (Cartagena, San Juan, Veracruz, Havana, Santo Domingo, Saint Augustine, etc,) or moved. Nombre de Dios was abandoned in favor of the more defendable Portobello. In 1607, the Spanish King ordered most of Hispaniola's cities destroyed and relocated to prevent illicit trading with pirates and smugglers (this is how the French got the western part of the island).