Modern people often have this image of the Roman state as being very legalistic, with clearly-defined laws, courts, and regulations ordering the state and offering justice to all citizens. Often this image is contrasted with the supposedly arbitrary justice of the Middle Ages. This view seems to be supported by two pieces of evidence. First, our knowledge of Roman court cases (many of which survive, in part, through Cicero), and second, the existence of the Digest, preserving the civil law and the juristic commentators on it. But the fact of the matter is that Roman law was not particularly systematic, the majority of Roman justice was summary, and at all periods of the Roman state the law was unequally enforced. The "systematic" Roman law was the criminal law, but criminal law almost only applied to crimes against the state or crimes committed by the politically and socially connected.
Quaestiones--courts--were convened to try criminal cases, but in reality the vast majority of crimes were not tried in a formal court with a selected jury and designated speakers. The magistrates generally in charge of most legal disputes were the praetors, most typically the urban praetor or the peregrine praetor, depending on the circumstances, or outside of Italy and the city then the provincial praetors (or whatever provincial magistrates were in charge locally). Within the city resided over a million men, women, and children, and it's simply implausible beyond belief that petty theft, murder, rape, arson, and so forth were not occurring every single day. The sources also back this up--the early emperors, for example, sometimes sat at the praetor's court or the tribunal to participate in justice. Now, many "crimes" were in fact civil offenses in Roman law (e.g. theft), but even so, the court cases that we know of are vastly insufficient to account for all the crimes that must have occurred at Rome. Murder, for example, was a criminal offense rather than a civil one, but murder trials were a big, complicated deal, and the only murder trials we know of are those involving important and well-connected people.
The impression we get from the Digest and our sources is that in fact in most cases the relevant magistrate, usually the praetor, would listen to any accusations brought before him and render a judgment on the spot. Most murderers, for example, would have been executed with no sentence other than the praetor's judgment, especially if it was an open-and-shut case. How did the praetors come to his conclusions? First, neither he nor any other state officials would likely even know about the crime until someone went to his tribunal and told him. Rome had no police force, no investigative body, and no public prosecutor. Bringing accusations to the judicial officials was entirely the duty of the citizens, and in earliest times most justice was itself the responsibility of the household. Once someone brought an accusation it was the praetor's job to investigate. The earliest preserved law, the lex repetundarum, already refers to the praetor "investigating" cases that have been brought to his attention, but there's no information whatsoever as to how this would have been done. With no forensic science possible, witness testimony was essentially the only way to verify events in the Roman world, and in most cases the praetor probably just asked the present parties what happened during the accusation and rendered a judgment.
Let's go back to your example of a merchant robbed by a highwayman or whatever. According to the Digest, furtum, or theft, was a civil offense rather than a criminal one. So right off the bat the responsibility for making a plausible accusation is entirely in the hands of our merchant. The jurist Gaius divided theft into manifest and non-manifest theft, that is theft in which a thief was caught with stolen goods or one in which he wasn't. Our merchant wouldn't have to catch the thief himself and present him to the praetor necessarily--according to the Digest, it doesn't matter who catches the thief, although there's some dispute about whether it matters if he's caught in the act or only later. If our merchant had witnessed the theft and had either apprehended or made note of the thief such that he could then be apprehended and arrested (by our merchant's dudes, most likely) to be brought in to the praetor, then he's in pretty good shape, although the Digest does note that in such an accusation you're on better grounds if you catch the thief before he can hide the stolen goods. There are other complications. For example, the Digest disputes whether an enslaved person can be considered to commit manifest theft, and the jurist Pomponius argues that someone hiding from a thief out of fear, cannot argue that the theft was manifest. You actually need to bring the thief in with the goods to make his crime manifest. Celsus, however, adds that if someone (doesn't have to be the person robbed) spots a thief in the act and chases after him, the theft is considered manifest if the thief runs, regardless of whether he's caught then or later.
Amusingly, non-manifest theft has a pretty simple definition: anything that's not a manifest theft. What this basically breaks down to, in most cases, is what sort of accusation our merchant is able to bring. Can he actually produce the thief in possession of the stolen goods, with witnesses attesting to the fact that the goods in the thief's possession belong to the merchant? If so, this is an open-and-shut case of manifest theft. If not, then more work has to be done. If, for example, our merchant only caught the thief's face and either recognized him or asked people who were able to identify him. Or if the thief robbed him in the night, and his only way of knowing who's responsible is that one guy's neighbors all insist he stole it. Or if our merchant brings a guy to the praetor or the urban prefect or the provincial governor and claims that this guy robbed him but has no stolen goods to show and only witness testimony? Clearly some kind of investigation, most likely in the form of cross-examining witnesses, is in order, but the Digest doesn't give guidelines on how to do that.
So what if our merchant is able to make an accusation against a particular person and the magistrate does judge him to have been the victim of theft? The penalty for theft differs according to the value of the object, the type of the object, the circumstances under which it was stolen, whether it's a manifest theft or not, whether it's theft or just fraud (the Roman definition of furtum is quite broad), etc. Manifest thieves were liable for four times the value of the stolen property, whereas non-manifest theft was punished only with twice the value. But things could get complicated. For example, a thief who breaks into your house and robs you while you're not there can be sued for four times the value of the property and an additional charge of non-manifest theft, even if he hasn't been caught with the goods.
Is it worth it? It depends. Merchants, in the Roman world, were neither laborers nor craftsmen, and therefore typically a cut above most common folk. With a reasonable degree of evidence and no good reason not to believe the plaintiff, Roman law seems to have believed the plaintiff over the defendant in most cases of petty theft. With a sufficient amount of evidence there would probably be no reason not to bring the case to the praetor, and even without a great deal of evidence if our merchant had someone against whom to bring a plausible accusation in most cases it would probably be in his interest to do so. Hence the case of a dude who breaks into your house while you're not there. Much of the Roman legal discourse about manifest theft seems to presuppose that the victim would take every measure to apprehend the thief and bring him to justice. If our merchant proved to be a lazy, greedy ass and just stood there instead of chasing after the thief or making whatever efforts were reasonable to identify and arrest him, the accusation probably isn't going to look good at the tribunal. Has our merchant got witnesses? Then he's probably in very good shape. If not, it's entirely his business to figure out more. Financially, however, if our merchant can make a plausible enough accusation to be worth bringing to the praetor, he should probably do so. The plaintiff could also demand back the stolen object, or compensation if it had been destroyed. However, Pomponius noted that a victim of theft had no right to demand compensation for any increase in the object's value after it was stolen. Finally, our merchant probably should consider whether his case is an honorable one of good standing. The Digest notes that only honorable accusations count. So, for example, what if our merchant is in debt, and an object that's been loaned to him gets stolen? In that case the person who owns it could bring suit instead. There's quite a lot of discussion in the Digest about these various circumstances, but the point is that complications--many of which would probably be relevant to a merchant, more so than a craftsman or a laborer--might give our merchant a bit of pause.