There were a lot of common clothing that Gould be user to distinguish not only legionaries but also members of the senate or general patriciuses or members of tha various noble classes (from equites to higher ranked families)

Togae for example were really common to distinguish not only senators but also members of the praetorian guard from normal legionaries (as soldiers from the legions)

This said everything the soldiers owned was given to them by the army but they had to pay for it and keep it clean and well mantained from the armor (lorica segmentata or squamata but also various degrees of chain mails) to their caligae(the shoes), the gladius, their pilum(javelin), shield (...) and their tunic and mantle

Military grade tunic was a brownish/dirty white wool tunic that soldiers wore all year and was thus suitable for every climate (soldiers in hot areas wore lighter tunics though)

This said you could distinguish a soldier principally from his shoes, his tunic (really different from civilian tunic for materials and colors and from senatorial grade tunics not only because paired with togae but also because those were expensive and decorated)

When off duty soldiers didn't wear armor generally because it was uncomfortable and heavy, remember also that it was forbidden to bring weapons inside Rome and generally inside cities and this was enforced by vigiles (you can thin of them as antiquate police )

However from the lowest official (optio) going up several degrees (centurion, prefectus, tribunus, legatus and so on) they were recognized for their grade and it was given them respect from everyone in the empire basically and they were recognized from marks on their equipment other than the vine stick typical of centurions

On the armor soldiers could also wear phaleraes that were metal disks representing the current medals

To make this short, it was a bit uncommon to see a soldier wearing armor off duty, it would be like seeing someone wearing a bulletproof vest today, it was also prohibited to wear weapons inside the biggest cities and Rome in particular so they would have been taken care of, except if they were officials, in that case they would have had the rank to overcome this

Soldiers with a rank had respect from others citizen because this mean that they sometimes also had a higher status than them

Tell me if I can improve something or if you'd like to know more about something I wrote, it's a broad topic