I have read at least five books on this subject and they all just skip right over this. They all start with how tensions were high and how the quartering act made most Charlestonians mad and there was a British appointed governor... and then suddenly the war has started and the British are trying to invade Charleston, being successfully repelled by the Rebels (and the palmetto logs) of Fort Sumter. It's like I'm getting an answer from someone on Seinfeld! They just yada yada yada'd the most important part! Were the soldiers driven out by torch-wielding mobs? Were tensions so high the British just left? I believe the governor snuck out of the city, which suggests there was some unrest. I've just never understood why the British would have given up this strategic port in the first place if they already had so many troops there.
I'm afraid the answer is disappointingly simple: there were no British regulars in Charleston in 1775, and by the time open fighting began, the British had other worries than immediate reconquest of a somewhat insignificant town.
The pre-war British army was quite small, numbering just under 50,000, and most were in Ireland or England itself at the outbreak of war. The vast majority of North American cities lacked British garrisons. There were not much more than 10,000 (I can't find an exact number) British soldiers in North America on the eve of war. The largest concentration was in Boston, where the British had been gradually building up troops since 1769 in response to growing tensions. Many of these regiments came directly from England or Ireland, but it also required the diversion of troops from New York (which was stripped to nothing) and Halifax (now Canada). The remaining British troops in North America were garrisoning modern Canada, the Caribbean, or former Spanish Florida, with a handful at frontier posts like Fort Ticonderoga in upstate New York.
By the by, the British bombarded and were repulsed by Fort Moultrie, an improvised fort on Sullivan's Island. Fort Sumter was built in the 1850s on an artificial island in Charleston harbor.