Apologies in advance, I'm cannibalizing an earlier post I made on this topic.

Basics of Mesoamerican Numbers

As others have mentioned, Mesoamerican cultures used a (edit:) vigesimal (base-20) number system. This means each digit in Mesoamerican numbers could hold a value of 0-19 with a new digit at 20 instead of 0-9 with a new digit at 10. There were two different ways to represent numbers in Mesoamerica. The first was bar-and-dot notation. This, like our number system, used a series of digits to express orders of magnitude. The numbers 0-19 were expressed as a combination of bars and dots. Each dot represented "1" and each bar represented "5". These numbers could be combined up to 19 (3 bars, 4 dots) and at 20 one moved to the next largest digit.

The second way of representing numbers was to use unique characters for orders of magnitude. So for example a pouch represents 8,000, so two pouches painted next to each other alongside four dots produces the number 16,004. The Aztecs used this system for most accounting purposes, and reserved bar-and-dot notation for mathematical calculations and carved stone monuments.

The bar-and-dot notation requires the use of zero. The earliest use of this number system appears to be a long count date recorded at the site of Chiapa de Corzo, and dates to 36 BC (Marcus 1976). This date does not include a zero, but since the number system requires a zero it's fair to assume Mesoamericans were using zero before the BC/AD switch.

Mathematical Operations

For sure, we can state that Mesoamerican cultures were capable of performing addition, subtraction, multiplication, and division. We know this because we see all of these operations used in the complex Mesoamerican calendar system, astronomical measurements, and architectural layouts.

The Calendar Round was a 52 year cycle used by (almost) every Mesoamerican culture. This 52 year cycle represented the alignment of two separate calendars, a 270 day calendar and a 365 day calendar. Arriving at 52 years as the interval at which these calendars intersect requires a knowledge of factoring, which in turn requires division. Saburo Sugiyama (2008) showed how all of the major buildings at the Classic Period city of Teotihuacan were built in intervals of 83 cm, which suggests a common unit of measurement. Furthermore, when the dimensions of those structures were divided by 83 cm, they yielded numbers significant in Mesoamerican religion and calendar systems, or multiples of those numbers. This is clear evidence of multiplication.

The use of astronomy also demonstrates a sophisticated mathematical knowledge. The surviving Maya codices include astronomical tables calculating the movement of Venus and Mars, which requires careful observation and mathematical calculations (Aveni 2001). It also appears that the Maya (and other Mesoamerican cultures) knew of the Eclipse season. That is, the window of time during which eclipses could appear in the sky. Now, it doesn't appear that they knew the world was round, or how big it was. So while they could figure out the eclipse season, they didn't know if an eclipse would be visible in their part of the world, or if it would be a partial or total eclipse. So it wasn't so much like they could say "today there will be an eclipse," but rather "if there's going to be an eclipse, it will be today."

More complex calculations

Although there is some evidence that Mesoamericans had some understanding of higher math, the evidence is fuzzy. It's clear they lacked some of the more complicated operations that we use today, but at the same time its hard to prove a negative because we don't have much to go on. We can say what they had, but not what they didn't have. Examples of Aztec geometry can be found in the Codex Vergara and the Codex Santa María Asuncíon. These documents include land surveys that record measurements of various agricultural plots in both dimensions and area (Harvey and Williams 1980).

Dimensions appear to have been measured using a unit of measurement known as a tlalcuahuitl or "land rod," which was equivalent to 2.5 meters. Measurements of area are recorded as well and appear to be represented as "square quahuitl" (Harvey and Williams 1980).

Area for rectangular plots appears to have been calculated the same way we calculate area today (multiplication of length and width). What's really interesting is how irregularly shaped agricultural plots were measured. They do not appear to have had trigonometry, and there's no recorded method for measuring angles. As a result geometric measurements were somewhat inaccurate. Aztec surveyors/scribes instead used more complex formulas for calculating area of irregular plots, such as averaging two parallel sides of an irregular quadrangle to get average measurements of that dimension.

Carmen Jorge and colleagues (2011) use modern geometry to assess the techniques used by Aztec surveyors in the Codex Vergara (following Harvey and Williams 1980), and conclude that about about 35% of the Aztec measurements of area are erroneous due to their apparent lack of trigonometry. Nevertheless, the remaining 65% appear to be accurate, at least based on the measurements given by Aztec scribes. Of the 35% "unfeasible" measurements, most (3/4) overestimate area by less than 10%.

Edit: here are some images linked from further down thread

• This map is one of the Acolhua land surveys, specifically lands in Oztoticpac. It demonstrates indigenous techniques for calculating area. The maps were drawn by indigenous officials. The writing in the margins was done by Spanish officials providing explanations as to what they say.

• Here is a figure showing the numbers 0-19 in bar-and-dot notation.

References:

• Aveni, Anthony. 2001. Skywatchers. University of Texas Press, Austin.

• Harvey, H. R. and B. J. Williams. 1980. "Aztec Arithmetic: Positional notation and Area Calculation." Science 210 (4469): 499-505

• Marcus, Joyce. 1976. "The origins of Mesoamerican writing". Annual Review of Anthropology (Annual Reviews) 5: 49–54.

• María del Carmen Jorge, Barbara J. Williams, C. E. Garza-Hume and Arturo Olvera. (2011) Mathematical accuracy of Aztec land surveys assessed from records in the Codex Vergara. Proceedings of the National Academy of Sciences of the United States of America. Vol. 108, No. 37 (September 13, 2011), pp. 15053-15057

• Sugiyama, Saburo. 2008. Governance and Polity at Classic Teotihuacan. Mesoamerican Archaeology. Julia A. Hendon and Rosemary A. Joyce (editors). Blackwell Publishing.