It very much depends upon the city, and the source describing the destruction.

In some cases, like Carthage, the destruction was severe. The Byrsa Hill - the centre of Punic Carthage that contained its most important public buildings - was literally removed and dumped onto the ruins of the houses around it, burying them (to future archaeologists' delight!). The Roman city was later built on top of this destruction layer.

But frequently, archaeology fails to find evidence of this kind of systematic destruction at sites that the writen sources clearly describe as being completely wiped out. In late antiquity, many texts describe repeated episodes where barbarians sacked and burned Roman cities, but evidence for this desteuction is rarely there. Gildas says that all the Roman towns in Britain were burned by the Saxons, but the archaeology shows they were abandoned over a few generations, not burned (when a city is burned, you get a layer of rubble that's really easy to spot when you excavate).

So 'razing' a city could mean a range of things, depending on the specfic detals of the case, and the point of the story that the author was telling. In cases like Carthage, it meant literally transforming the civic centre of Carthage into rubble (though salting the earth was a 19th century embellishment). But it was also a literary trope that could serve a variety of functions ranging from exageration to invention to make the story more exciting or compelling. The razing of a city might mean 'killed, raped and enslaved a lot of people,' or just 'there was a battle nearby and know nobody lives there, but the story is better if everything catches fire so, sure, that totally happened. The city burned to the ground.'

In the worst case scenarios like Carthage, how was it done? The same way the Romans built roads and bridges: picks, shovels, and an army of men used to hard work. The Roman army was as good at moving soil as they were fighting; digging ditches and moving earth (usually constructively) was one of the ways their officers kept them busy when they weren't fighting.

For example, a city like Carthage. After the Third Punic War, it's claimed that the Romans razed the city. But yet, Carthage was then occupied and became an important city in the Roman Empire.

The original city of Carthage was burned to the ground, the buildings reduced to rubble, and tens of thousands of its citizens were taken into slavery.

After the fall of Carthage, the nearby city of Utica was made Roman capital of the region. Later, that city's harbour was rendered useless due to silting caused by agricultural run-off, and the Romans decided to rebuild Carthage to reestablish it as capital of the African province.

So they did not raze it with the intention of moving into the area and rebuilding it, and they did not occupy the area immediately after the Third Punic War and the Battle of Carthage as your question supposes.

To answer your question more broadly, when it is said that a city was razed to the ground (as opposed to occupied or sacked), it means that it was utterly destroyed. Often, the inhabitants of the conquered city were commanded to assist in the city's destruction. Typically, it was looted, its citizens killed or taken as slaves or disbanded. This was often done to ensure that no one would occupy the area in the future. Great cities, walls, and palaces were symbolic of the power and wealth of the kingdom, and their destruction was symbolic of the superiority of the conquering civilization and their Gods. Often, monuments were constructed of debris to mark the place.

Cities meant for occupation weren't typically razed, of course. Razing was the intentional and deliberate destruction of an enemy city or state for strategic, political, and or economic reasons. Sometimes it was simply for spite because the conquerors really, really hated their enemy.