To answer your opening questions, there was no state-run debtor's prison in Republican Rome (or really any state-run residential prisons as we think of them). Caesar also would not, by force of law, have been exiled, although he may have had incentives to flee Rome as discussed below.
To understand what would have happened, it's important to discuss a little bit of history. In the early republic there was an institution of quasi-debt-slavery called nexum; it was primarily a private process that did not require the intervention of a magistrate--once certain conditions were met, the creditor could simply "lay hands" (manus iniectio) on the debtor to effect the nexum. This feature made the process both liable to abuse and deeply unpopular. Later reforms eliminated the "self-help" features of the debt collection, leaving an unsatisfied creditor the option of going before a magistrate (the praetor) to seek a judgment.
The laws about what happened next were very different at different times, so what I'll describe next is specific to Caesar's period, except where otherwise noted. In Caesar's period, if the judgment weren't satisfied within 30 days, the creditor could bring the debtor before the praetor for non-payment. If the creditor prevailed in proving non-payment, the creditor then had two options:
He could (with judicial sanction) "lay hands" on the debtor and imprison him for 60 days. During that period he had to exhibit the debtor on three separate market days, to allow someone, presumably his family or friends, to pay the debt on his behalf (and arguably to show proof of life). If there had been no payment in 60 days, the creditor could hold the debtor as long as it took the debtor to either work off the debt or otherwise satisfy the debt. As an aside, in the early republic, the creditor could sell the debtor into foreign slavery or execute them after 60 days, but this was no longer the case by Caesar's day, the worst Caesar would have faced was being held until he satisfied the debt either through labor (unlikely given the size of the debts) or through friends and family (more likely).
By the late 2nd century B.C.E. the limits of option 1) became clear: if you couldn't find the person to imprison them, a creditor was left without recourse. So the praetors developed an alternative method (bonorum venditio, literally "the selling of goods"), which involved the attaching and selling the debtor's property, which is analogous to the modern bankruptcy system (it actually provided a model for the modern bankruptcy system in much of the world). The praetor attached or seized the debtor's property to prevent the debtor from selling or hiding it; public notice was given to allow other creditors to come forward; and then the property was sold and the proceeds divided pro rata among the creditors. A key difference must be noted between the modern system of bankruptcy and the Roman system in Caesar's day: in modern bankruptcy after the division of the estate, debts are typically discharged (i.e. an individual no longer owes the money after going through bankruptcy); in Caesar's Rome the debtor remained liable for any unsatisfied debts after the sale, and additionally could incur a negative legal status called infamia ("infamy"), which had various legal consequences (most salient to Caesar would have been a prohibition on infames holding public office). The sources suggest that creditors would in some cases waive their right to imprison the debtor in order to pursue the venditio, but it is unclear if this was a mandatory part of the procedure or a gesture of goodwill; it is at least theoretically possible that a creditor could seek a venditio and then subsequently imprison the debtor on the basis of any unrecovered debt.
One of the reforms instituted by Augustus was to allow debtors to voluntarily give up their property and participate in a venditio like procedure (cessio bonorum, a "yielding of goods"), which was paired with an actual discharge of all debt, but that only came later. In the later imperial period, the ability to physically seize the debtor fell into disuse and was abandoned, while the bankruptcy-like procedure became the normal procedure.
So as you can see, Caesar might have had good reason to flee Rome; if he stayed, he rendered himself liable to imprisonment for his debts, potentially in addition to a judicial sale of all his property, whereas if he left, he faced only the latter.
Sources:
Andrew Borkowski's Textbook on Roman Law
Hans Julius Wolff's Roman Law: A Historical Introduction
EDIT: Fixed some typos.