Just looking at historical populations Charleston was right up there with those cities until the 1820s. Obviously the industrial revolution played a huge part of that, but why didn't Charleston experience the sort of immigration and explosive growth as some of its northern counterpoints??
Charleston was mainly a port for a agricultural products grown using slave labor in the south, pretty much up until the civil war. While it was an administrative and cultural center, the vast majority of Charleston's wealth was based around international trade of rice, indigo, and some cotton. This required a number of international traders, bankers, and shippers to be based in the city, but the factories that drove the massive growth like that seen in northern cities never cropped up.
A much larger historical question is why the south didn't experience industrialization - there's a lot of debate on that question. However, whatever the reason, growing cities in the 1800s required industry to expand their populations and attract immigrants. Charleston never developed these to the same extent.
Source: Plantation Economy in South Carolina by Edelson S. - covers the development of the economy in South Carolina.