It was geometry.
I'll try to answer this question with respect to physics because I'm not a historian of mathematics (sadly there's not enough of them around, maybe I should become one). Basically all formal mathematics in the Ancient World was what we would recognise as geometry. Certainly whenever ancient scientists^1 tried to analyse the world mathematically, geometry was their weapon of choice. If you read the books of someone like Archimedes (287-212 BC), they're basically just collections of geometric proofs, yet they're proofs designed to talk about the natural world.
If we skip forward quite a bit and start looking at someone like Galileo (1564-1642), we see essentially the same tradition, albeit in a significantly more reader friendly version. His final book, Discourses and Mathematical Demonstrations Relating to Two New Sciences (1638) focuses on physics (the two new sciences in question being the science of materials and the science of motion, you can read it here, it's fun to flip through), and it's structured exactly as its title suggests. The book focuses on three men who are having a conversation about a variety of topics concerning matter and motion. The men talk for a while, argue back and forth, and then eventually one of them (usually Salviati, who is Galileo's mouthpiece) will say "and here our learned friend has clearly demonstrated this, as we will now show", and the text will become a series of mathematical proofs. And the proofs are all about geometry.
So that's the situation in the early mid 17th century. A generation later the same basic structure is in place. The problem is that the subject has developed quite a bit since the 1630s, and the mathematics has become more complex. And I really mean a lot more complex. Those images are from Christiaan Huygens' (1629-1695) Horologium Oscillatorium (The Pendulum Clock, 1673), and are not isolated examples. Most of the proofs in that book are that complex. If we look a little further into the future, we get Isaac Newton's (1642-1727) Philosophiæ Naturalis Principia Mathematica (Mathematical Principles of Natural Philosophy, 1687). In it you will find proofs that are similarly complex. Despite being famous for the [co]invention of calculus, Newton's magnum opus is written in the language of geometry, not calculus, probably because Newton felt it would be too much to introduce a major new theory in a totally new language. Many of the proofs in the book Newton had in fact originally discovered via calculus, he simply translated them back into geometry.
The point is that by the late 17th century the geometry underpinning mathematical physics had become extremely complex, and with that complexity came unwieldiness. The geometry that Huygens and Newton used was so complex that even their colleagues struggled to understand it. Edmund Halley (1656-1742) found Newton's mathematics so complex that he had to ask Newton for help several times, eventually prompting Newton to write out his theory in full, which became the Principia. That's Halley of Halley's Comet by the way, he was no slouch. The fact that the geometry of physics had become so difficult to deal with in the late 17th century prompted people to look around for alternatives, and when they couldn't find them, invent them themselves. So you have Newton and Leibniz inventing calculus at pretty much exactly the same time independently of each other, because they're both trying to find a solution to the same problem. After about a generation it was clear that calculus was far simpler and more powerful than the old style of mathematics, and physicists abandoned geometry basically entirely and never looked back.
I'd like to put a mathematician's spin on it.
"Mathematics," meaning "that which is learned" is believed to have been coined by Pythagoras of Samos around the 6th century BCE. (Pythagoras is also said to coined the term "philosophy" - "love of wisdom.") Prior to that they did arithmetic and geometry. Note that they were not alone in that, the Egyptians were also doing arithmetical / mathematical things, as were the Babylonians and others.
At about the same (give or take a few centuries) the Mayans were developing their vigesimal (base 20) mathematics. Of course none of that contributed to today's mathematics but as it was mathematics and before the calculus it should be mentioned here.
Also in that rough era the Chinese were doing some interesting things. Using a polygon of 192 sides and applying the method of successive approximation, Liu Hu calculated the value of pi accurately to five decimal places.
Long before Pythagoras, in the 7th or 8th centuries BCE, the Indians were circulating a text called “Sulba Sutras” which had some Pythagorean-like things. Some people claim it is where Pythagoras, on Samos just off the coast of what is now Turkey, learned basic geometry. Additionally, the Sutras give geometric solutions to linear equations and quadratic equations in a single unknown, and more.
"Greek" (there was such thing as Greece nor Greeks at the time but it is convenient to call them that) mathematics was the All Geometry, All The Time, channel. The sage Thales, sometime prior to 550 BCE, set out to develop geometry as an abstract discipline. It is generally believed that he was the first to do so; he is the first we know of to do it.
One of my undergrad math profs said something in our mathematical logic class that I can't verify, though it has every appearance of being correct. He said the Greeks thought of numbers as quantities (of things), the product of two numbers is area, the product of three numbers is volume, the product of four numbers was nonsense. Although we don't if that is strictly true - it's not written down anywhere that I know of - what we do know of their geometry and arithmetic supports the idea.
So back to Pythagoras. What Pythagoras did (and is believed to be the first but again, there's a helluva lot of that age we don't know much about) was to make a system of mathematics wherein numbers correspond to the geometric elements.
Around the middle of the fifth century BCE Hippocrates of Chios assembled his book The Elements. It is the first known comprehensive compendium of the elements of geometry.
Over the next few centuries they got into some snags - things like infinity and irrational numbers (numbers which can not be expressed as fractions). Irrational numbers in particular was a thorn in their sides because their concept of numbers was so closely tied to geometry which is constructed only from rational numbers. There's an apocryphal but amusing story about the discoverer of the proof for the irrationality of the square root of two. The story goes that the guy who discovered the proof did so while out sailing in the Mediterranean. When he showed the proof to his shipmates, they threw him overboard.
It was during that era that the Greeks made a really huge contribution to mathematics, namely the deductive proof. The Egyptians and Babylonians used an inductive method to establish the truth of a "theorem." Post Aristotle, the Greeks had a rigorous system of logic (which was used for about 2000 years) by which they could proceed from axioms, using deductive logic, to produce theorems.
Circa 300 BCE this guy named Euclid came around. You've probably heard of him. He took everything that had preceded him and put it into his book Stoicheion, "Elements.” He didn't just reproduce the hundreds and hundreds of theorems and proofs that his predecessors produced, he explained them clearly, logically, and we mathematicians like to say, elegantly. And he did it all using only a compass and straight edge. Perhaps the most important aspect of Euclid's work on geometry (he did other stuff like optics too, and more) was that he reduced all of geometry to five axioms (some prefer to call them "postulates"). A postulate or axiom is something that is so evident that it is taken to be true. Four of Euclid's axioms fit that description to a T. The fifth one though, that caused some head scratching. Known as the parallel postulate, it is not anything like self evident. It is also far more complicated the first four which are each expressed in less than a dozen words - some of them are only five words long. It wasn't until well into Elements that he even drew upon the axiom, in book 1 proposition 29 (or was 27?).
Euclidean geometry is still valid and is widely used even today. It's not terribly pertinent to this discussion because Bolyai and Riemann and Lobachevski didn't come up with non-Euclidean geometries until the 19th century, well after Newton (pace Leibniz) introduced the calculus, but it's worth noting that until they did people were still scratching their heads over the fifth postulate. Countless hours of effort had been spent over the millennia in attempting to make the fifth postulate a proposition (theorem), proved using the other four. Nope, no one ever did.
Just for completeness sake, the fifth postulate can be stated thusly: Given a line A in the plane and a point not on the line, there is exactly one line B passing through the point and parallel to the line A. In the Bolyai Lobachevski geometries an infinite number of parallel lines pass through the point while in Riemann's there are no parallel lines through the point.
Not a whole lot happened for the next thousand years or so.
In the 9th century an Arabic fellow named Al-Khwarizmi did a lot of work and basically invented algebra. The name itself is from the Arabic. A LOT of that filtered into Western (European) mathematics and was further developed both pre Newton and post Newton.
Between the 4th and 12th centuries nothing happened in Europe. Nothing mathematical worth speaking of, anyway. A joke should go here, something about the lack of brilliant mathematicians is why it was known as The Dark Ages.
I have to go make dinner soon so I'll try to wrap this up.
Not a whole lot happened for the couple hundred years. In the 16th century Cardano published Ars Magna in which he gave solutions to cubic and quartic equations. But at that time negative numbers were generally disregarded. Cardan said they were roots of equations but in his view they were impossible solutions, fictitious. Cardan also introduced what would become upon further development complex numbers.
A bit later on Descartes partially accepted negative numbers but still fretted about complex numbers as roots of equations versus things meaningful in themselves.
We are now nearly upon Newton so I'll go make dinner.
Edit: tpyos
Followup question: What happened in math/physics besides geometry?
I remember Oresme proved the divergence of the harmonic series, Fermat proved number theory stuff, and the Islamic world did algebra. Surely the academic world consisted of more than just geometry?
