Romans did not use “Roman numerals” for advanced mathematics.

Firstly, educated Romans spoke Greek and the vast majority of serious Roman mathematics were actually written in Greek. The most important work of mathematics by a Roman was easily Diophantus’s magnus opus Arithmetica, written in Greek. Other examples include Catoptrics (attributed to Euclid but probably written by Theon of Alexandria), Ptolomy’s Almagest, and Boethius’s De arithmetica. This is true in the classroom as well. We have educational materials, probably aimed at advanced students, written by Theon of Alexandria which are also in Greek.

Secondly, there were multiple number systems in widespread use (at least among mathematicians) at the time. Babylonian arithmetic was a place-value system like our own, but in base 60 instead of base 10. The Greek system wasn’t a place-value system per se, but doing arithmetic by hand in it is not too challenging. Both of these ways of representing numbers see much more use in Roman mathematical texts than Roman numerals.

Thirdly, even if someone wanted to do arithmetic “in Latin” they wouldn’t necessarily do the kind of symbolic manipulation that we are used to today. Arithmetic was typically done via abacus, a mechanical device that assisted in computation. “Latin” “Greek” and “English” abici are all the same – the system you use to write down the result in no way effects it’s operation.