To start with, you probably don't wear your toga to a bad neighborhood.

The toga was not everyday Roman wear; it's more the equivalent of the tuxedo. For an everday garment, it is in fact tremendously impractical and cumbersome: it's expensive to make, and still more expensive to keep clean, and hot to wear, and uncomfortable, and awkward and restricting to move in without it falling into disorder. A Roman had to have some practice wearing it just to put it on and walk around in it without looking like a moron -- some would say intentionally so, because that makes wearing it properly a sign of good breeding. Not unlike wearing a tuxedo properly.

Correspondingly, during the late Republic, the proper toga started falling into disuse, replaced most of the time by a Greek-style cloak or pallium, basically just a small blanket that you throw around yourself. Augustus, ever conservative (once he took power), evidently annoyed at seeing people dressing down, made it mandatory for citizens to wear the toga in the forum. (He may also have ordered that people in dirty or unorthodox-colored togas couldn't sit in the front rows in the theater.)

But let's say that you decided to wear your toga-tuxedo on a night-time stroll through the suburra. Well, if you're wearing a full toga on a stroll around town, you probably have a small troop of slaves working as your bodyguards, so hopefully you're not going to have to do anything physical yourself. A street brawl would be beneath your dignitas, after all.

But let's say, again, that your slaves have taken the night off (maybe it's Saturnalia), so you find yourself alone when you get jumped in your tuxedo toga. Well, you're in trouble. Your best bet is to take a few seconds before the melee to hitch it up. Plutarch, in the Life of Tiberius Gracchus, describes the senators girding up their togas before the big kerfuffle in the forum -- I've always imagined this like samurai using tasuki to tie up their wide sleeves before activity. But the fact that this was the senators' immediate reaction when they knew that things were about to go down suggests that, if you have to fight in a toga, tying it up in some form is ideal.

If you don't have time for that, you're probably screwed. I would think that, if you don't have time to hitch it up, you don't have time to get out of it either, since it's wrapped around you so many times. Your best bet is probably to just leap into the melee without a thought of your garment. But then it's likely to come unraveled, which would leave a nice hold for your enemies to grab.

TLDR: don't wear a toga to a bad neighborhood when your slaves have the night off.

Followup question: how often would a person rich enough to afford a toga wander into a bad neighborhood?