Regarding weapons in the city, I'm afraid your premise is quite wrong. Here is a previous answer from /u/Zaniac0 which clearly shows that people had easy access to weapons inside Rome, and that indeed there were gangs of armed thugs who waged what amounted to urban warfare upon each other. It was fairly easy for anybody with money to scare up a sort of informal brute squad, but the utility of this was somewhat limited because whoever you wanted to thump could probably get one too (as Cicero's life was apparently saved by his own rent-a-goons in the story from that comment.) I'm not sure whether people ordinarily carried weapons around as part of day-to-day life, but they could certainly get them and they certainly knew how to use them, or for the upper classes, how to hire people who could.

But ok, suppose you wanted to loot the Temple of Saturn (where the Roman treasury was located), or maybe some less prominent temple, and you hired a gang of goons and you successfully invaded the place, killed all the priests and whatever guards they had, and broke into the treasury. Congratulations! All you can do is sit down and cry, because you forgot the most important thing about large quantities of gold and silver: they are really damn heavy. A Roman Talent (the typical measurement for money or bullion in bulk) weighed 71 lb (32 kg), so each of your guys can carry maybe 2 or 3 talents if he's brought a sturdy enough backpack or, better yet, a wagon. Or lots of wagons. This, however, is peanuts compared to what's in the vault; there are hundreds or thousands of talents of gold or silver there.

The Romans had a very advanced and complex banking system and they did most of their big transactions on paper for this exact reason; it simply isn't practical to move large quantities of gold around without a military escort and heavy-draft hauling gear.

But if you're determined to continue, you'd have to carry your heavy load (or guide your wagon train) all through a huge city full of citizens who are likely to, ahem, remonstrate with you for *literally stealing from the gods.* Which, by the way, you're rather unlikely to have been willing to do in the first place; it was hard to find people who were willing to piss off the gods on purpose in ancient Rome. Anyway, unless you managed to stone cold spec-ops it and kill all the people in the temple without a sound, word is going to spread. Even if you didn't steal enough to cause the militia to be called out, as above, there are lots of gangs and freelance goons around. In other words, if you intend to carry a heavy load of gold bullion either on your back or by wagon through the streets of the violent and chaotic city of Rome without being stopped either by citizen do-gooders looking to score some points with the gods, or other entrepreneurs like yourself who would like a piece of the action, you can save yourself a lot of time by simply cutting your own throat.

Now, there was a legend in Rome about a big heist, but it wasn't in Rome. Basically, a guy called Servilius Caepio was generaling his army around Gaul and he somehow obtained an immense amount of money: 15,000 talents (or 825,000 lb) of gold according to one source. The ancient sources are very vague on exactly what happened, but basically, the money was supposed to be sent to Rome, but never got there. Caepio was put on trial for this, but got off the hook. Then he was put on trial again for getting his ass very thoroughly kicked and his whole army wiped out by the Germans (who were at that time causing no end of worry and bother), and coincidentally fined 15,000 talents (which was quite a bit more than was in the Roman treasury at the time) and exiled from the city for life. One historian, in a dramatic manner, tells us that Caepio was considered to be cursed because the gold had been stolen from temples and somewhat gleefully tells us that Caepio died in misery and his daughters had to become hookers. At this point I refer you back to what I said earlier about stealing from temples; even if you weren't scared of the gangs, everyone in the ancient Mediterranean basis is hella scared of the gods.

TL;DR: Ancient people would probably not consider doing this for fear of divine retribution, but even if they did, it's not logistically feasible to transport large and heavy loads of stolen money through a lawless city in secret.