Archaeological evidence suggests a pretty simple answer: Fire. Cities that were destroyed often have what's called a "burn layer," a layer of sediment between the stuff that was built before and the stuff built after. Fires were set and the population of the city prevented from putting them out. For example, Jeremiah 52.13 says that the Babylonians “burned the house of the Lord, the king’s house, and all the houses of Jerusalem; every great house he burned down,” which may have just been confirmed by burn layers discovered in the City of David.

Another technique was mass deportation. The population of a city would be uprooted and transplanted somewhere else, sometimes with a different population put in their place. This is what happened, for example, in the northern kingdom of Israel: When the Assyrians defeated the Israelites in 722 BCE, many of the inhabitants of the capital city of Samaria were transported elsewhere, becoming the "Ten Lost Tribes." That might not seem all that destructive, but consider that the Samaritans, descendants of those left behind, are still not considered real Jews by the state of Israel. Mass deportation is effectively genocide, destroying the city by destroying the identity of its inhabitants. See Oded, Mass Deportations and Deportees in the Neo-Assyrian Empire.

As for Carthage, legend says that Scipio had the fields sowed with salt so that nothing could ever take root in Carthaginian soil again. While sowing the fields with salt was a well-known ritual in the ancient world, this is pretty unlikely: The amount of salt that would have been required would have been astronomically expensive, and Carthage was in fact later re-inhabited. Instead, the city was set on fire and its population deported (i.e., enslaved). See Ridley, “To Be Taken with a Pinch of Salt,” Classical Philology 81.2 (1986).

Edited: Expanded.