Short answer:

Likely empirical analysis, rules of thumb, rather than any complex calculations

Discussion:

We wish we knew more than we do about financial analysis in antiquity. We have very little in the way of supporting documentation for things like loans and mortgages -- and essentially no direct evidence of things like banks' capital ratios.

We do have _some_ evidence for borrowed funds as a source of capital, Morris Silver notes:

Plutarch mentions taking loans to purchase slaves, fields, vineyards and olive groves ( Moralia 523f, 830e). The lecturer Dio Chrysostom (46.9, 47.21), a contemporary of Plutarch, had the collection of rent in mind when he borrowed funds to construct a kind of shopping area near the baths of Prusa (Bithynia). Legal references in the Digest show us instances where loans were contracted with the objective of buying some land, building, equipping or purchasing a ship, repairing a ship, feeding sailors, purchasing merchandise (e. g. marble) for resale and to a slave for running an oil business

We wish we knew who the lenders were here-- were they bankers? Wealthy private individuals? We don't know. How did the lenders account for these loans? We don't know.

Modern economic historians have attempted to reconstruct what credit facilities may have looked like using the available evidence. Two who come to mind are Peter Temin and Cameron Hawkins -- both of these scholars are inferring a lot by looking at the kinds of transactions and businesses we see in Rome, and then comparing these businesses to better documented businesses later and calculations we can perform today.

What we can say is that in antiquity a lot of things that we do by analysis were done ad hoc and by "rules of thumb" rather than calculation. We can see this in things like architecture-- the Romans couldn't do finite element analysis or other structural engineering calculations, rather they learned from experience "what arches stand up, and which fall down". This has been discussed on a number of earlier threads here, on the topic How did roman architects and builders calculate ?

We can see reasonably complex financial instruments such as the "maritime loan" documented in the Muziris papyrus (from the 2nd century CE). This document had elements of both an insurance contract _and_ a secured loan, and today would have to be analyzed as a hybrid instrument, an insurance analysis for that component and a loan analysis. A modern lender/insuror would also have to do complex calculations to decide whether they had the capital to support such an exposure . . . there's no evidence that any such analysis was done.

Although there have been a number of brave attempts to impute a contemporary free-market philosophy to the ancient word, these seem a bit optimistic to many, including me. We are missing evidence for a lot of basic macroeconomic analysis in the ancient world:

However, it·has been observed that not one ancient commentator, no matter how "attentive to the particular circumstances in which he found himself from day to day, or which he described as an historian, offers any reflections on what we call long-term movements, on the secular movement of prices". (Finley, p.142)

So how did people decide how much they could lend? Likely the same way they decided how to build buildings-- though trial and error and experience with "what structures stand and which don't", which businesses make money and which don't. This would mean that Roman financial structures were likely overcapitalized, just as Roman architectural structures were overbuilt-- when you can't calculate "what is safe" . . . you have to provide a very generous "margin of safety" otherwise your structure collapses.

We have unhappily little evidence for any calculations of, say, capital budgeting by private citizens in antiquity-- in Rome or elsewhere. To the extent that we do have information, the impression that we're left with is that the State was a significant player in absorbing capital expenses, with merchants piggybacking on State infrastructure and subsidies, rather than taking on the capital risk themselves. So, for example, the State would contract with merchants to build vessels to haul grain -- and the entrepreneurs who won these contracts might then piggyback some of their own cargo on these vessels. Such transactions moved the capital risk to the State -- the merchant didn't have to fund the construction, calculate a payback period, or secure a loan. Rather they just had a much simpler transaction "maybe when we're in Egypt we buy some honey as well as our grain cargo and sell it when we get back to Rome".

See:

Cameron Hawkins “Artisans, Retailers, and Credit Transactions in the Roman World”, Journal of Ancient History 5 (2017), 66-92

Finley, Moses. "The Ancient Economy" (University of California Press:1973)

Peter Temin, "The Roman Market Economy" --

Casson, L. “New Light on Maritime Loans: P. Vindob G 40822.” Zeitschrift Für Papyrologie Und Epigraphik, vol. 84, 1990, pp. 195–206.

Silver, Morris. “FINDING THE ROMAN EMPIRE'S DISAPPEARED DEPOSIT BANKERS.” Historia: Zeitschrift Für Alte Geschichte, vol. 60, no. 3, 2011, pp. 301–327.

Bromberg, Benjamin. “Temple Banking in Rome.” The Economic History Review, vol. 10, no. 2, 1940, pp. 128–131.

Bromberg, Benjamin. “The Origin of Banking: Religious Finance in Babylonia.” The Journal of Economic History, vol. 2, no. 1, 1942, pp. 77–88.

Oldroyd, David. “THE ROLE OF ACCOUNTING IN PUBLIC EXPENDITURE AND MONETARY POLICY IN THE FIRST CENTURY AD ROMAN EMPIRE.” The Accounting Historians Journal, vol. 22, no. 2, 1995, pp. 117–129.

They knew the math and calculated interest / risk just as we do today. They typically used back of envelope calculations and there's even evidence of people using compound interest calculations dating back to ~1000 BC. Compound interest sounds sophisticated but it's really quite simple and is even used by cow herders. One ancient document records that a person wanted to borrow the cow herd of another for a season. Because cows beget more cows, and the longer you borrow them the more cows are begotten, the document lending the cowherd from one person to another included that the original owner expected more calves the longer the herd was rented, and the more of them which were rented. The rental agreement had calculations based on the rule of two, IE every two cows was expected to create one more, and then after a season the calculations were repeated to account for contingencies. That's compound interest, just in cow form.

There were companies that provided shipping insurance, and there were rough equivalents of limited-liability companies in which the owners were held liable only for the amount of capital they put into a project. Typically these entities were formed on a per-project basis.

All these things combined to create a pretty modern-sounding system. For example, 5-6 Greek businessmen in 1500 BC might go in together to buy Greek olive oil, pay a boat captain to ship it to southern Ukraine, and insure their shipment via an insurance company. They'd then trade the olive oil for furs or precious metals in Ukraine, separately insure that cargo, and ship it back to Greece for sale on Greek markets. Sounds pretty modern to me. The businessmen could even borrow (albeit at high rates of interest) to get the capital for their stake.

All of this happened in the ancient world. Before the collapse of the late Bronze Age civilizations c. 1180 BC, there were sophisticated trade networks linking Italy, Greece, Egypt, Turkey, the Levant, Afghanistan, Cyprus, and Ukraine.

Many of the oldest extant written documents are financial or trade documents, varying from simple receipts ("X received 30 talents of copper from the merchant Y") to documents exempting certain merchants from having to pay taxes.

Markets, finance, and civilization all go together.

Source: Money Changes Everything: How Finance Made Civilization Possible by William Goetzmann.