This article looks as if it might answer your question.

Dark cloaks are a practical choice. Far less cumbersome in handling than a toga and you don't immediately see every fleck of dirt on them as you would with white clothing.
People trying to dress casually even for social functions (like the theater, the forum, ...) is of course understandable but for Augustus the white toga seems to be a matter of dignity and Roman identity. He wouldn't have complained about dark cloaks so much if people generally had shared his opinion.

I don't think it's about sketchiness in the sense of "robbers wear dark cloaks at night so they can't be seen".

Pallium (Palla for women) was an adaptation of the Greek himation. It was a multi-purpose clothing object, it could be used as a blanket, covering yourself after a bath, protection for the weather. It was fastened by a fibula, a type of brooch. It was the most common piece of clothing of the age.

However, in Roman society citizens were expected to wear togas in public places such as the Forum - it was looked upon as a sign of citizenship. The way that Romans wore the toga though - loosely, over a tunic - made it impractical, and by the time of Augustus it seems that it was going out of fashion.

Cicero would often use a manner of dress to attack people in his speeches. In the Second Speech against Catiline, he says, speaking of Catiline's friends "These are the men you see with their carefully combed hair, dripping with oil, some beardless, others with shaggy beards, with long-sleeved and ankle-length tunicas, and wearing vela, not togas. All the activity of their lives and all the efforts of their waking hours are devoted to banquets that last until dawn. In this herd you find all the gamblers, all the adulterers, all the filthy minded lechers." (from "The World of Roman Costume" by Judith Lynn Sebesta, 2001., quoting Cicero "In Catillinam" 2.22)

Augustus reportedly quoted Virgil "Romanos, rerum dominos, gentemque togatam" ("Romans, lords of the world and the toga-wearing race"), when he saw people wearing dark cloaks in the forum, and proclaimed that no one should be allowed in the Forum except in a toga and with no cloak, and that at public games, dark cloaks would only be allowed in back rows (along with the women). He also put the Senators in front, separated soldiers from the civilians, and made separate accommodations for Vestal Virgins so that they sat facing the praetor's tribunal, just so you don't think he was especially concerned with just the cloaks when it came to public places.

Whether there was something significant about dark cloaks in particular, I can't tell. Maybe darker colors were fashionable or maybe it was seen as something a hoodlum would wear in the dark streets at night. Either way, the main problem that conservatives had with cloaks was that they weren't considered proper public dress for a Roman man.