I was reading Sokal & Bricmont's Fashionable Nonsense when this occurred to me.
By state of the art of Mathematics, I mean not only theoretical and applied mathematics, but also history and philosophy.
It's been some years since I've read the Manual Manuscripts, but I remember some… unorthodox ideas about Calculus, at least from a modern standpoint. I was curious whether Marx's ideas fall in line with those of his time
Epsilon-delta proofs were pioneered by Weierstrauss and Bolzano, Bolzano gave an epsilon-delta definition of a limit in 1817, and Weierstrauss gave the definitive modern definition a few years later. When Marx wrote his manuscripts on infinitesimal calculus, it was not gospel that epsilon-delta definitions were the way to go (he was writing in the 1870s and 80s, which while long after Weierstrauss, was during a time period when infinitesimals or fluxions were still used by mathematicians). In fact in the 19th century there were many competing approaches to understanding infinitesimals.
Infinitesimals have in fact enjoyed a kind of return to fashionability in this century. Hyperreals and smooth infinitesimal analysis offer two competing accounts of how we can rigorously do mathematics with infinitesimals.
In the philosophy of mathematics, two extremely important works were published around the same time as Marx was writing. Frege's Foundations of Arithmetic and Husserl's Philosophy of Arithmetic represent two very major salvos in a fight over the foundations of mathematics which has since been discussed at length by historians of mathematics and philosophy (I can give recommendations for that literature). Iirc both mention infinitesimals off-handedly, but with an attitude that 'we can't even figure out the natural numbers yet, but once we do we can give a rigorous basis for the rationals, reals, complex numbers, and infinitesimals'.
The foundations of analysis in general were a major topic of that time period. Dedekind did some pioneering work towards giving the first model of the real numbers using the notion of a Dedekind cut. Dedekind published this work in 1872. Marx would have been writing in that milieux.
If you have more specific questions I can try to answer them, mathematics was a bigger field back then just as it is today.
Also Sokal and Bricmont are generally not reliable, and reading them is liable to mislead more than it is to enlighten about the reception of mathematics and science in continental philosophy.