The key measurements are the positions of planets relative to stars (i.e., the "fixed stars") or fixed points or lines (like the zenith, meridian, horizon), and stars relative to fixed points or lines.
These measurements are measurements of angles - the angle between lines of sight from the observer to the observed objects and the reference object/point. You can make a simple instrument for such measurements with a protractor (so as to have a scale for the angle measurement) and two arms/rods/sticks with sights on them to line up with the observed/reference objects. The two arms pivot about the centre of the protractor. Line up the sights with the objects, and read off the angle between them. Depending on the maximum angle that such instruments can measure, they are called quadrants, sextants, etc.
So the question is: how can such an instrument be made more precise? The accuracy with with the arms can be lined up with objects depends on their length. Therefore, the single most important thing for more precision is
However, large instruments mean that more care must be taken with mounting them so that they are stable, yet still moveable, and the parts must be thick enough to be rigid enough so that bending of the arms, etc., under their weight doesn't introduce excessive error.
The largest instruments could be very large. The very large ones were fixed, typically in the north-south direction, and were used to measure the altitude (i.e., angle above the horizon) of stars/planets as they cross the meridian. For example, al-Khujandi built a mural quadrant ("mural" = built as part of a wall, "quadrant" = measures up to 90 degrees) with a radius of 20 metres, near modern Tehran, and Ulugh Beg built a 40 metre instrument of the same type in the 15th century.
Instruments that could be used to make measurements at any point of the sky needed to be smaller, so that they could be pointed in any direction. Tycho Brahe's instruments like this typically had radii of 1.5 to 2 metres, and delivered accuracies of about 1/2 a minute of arc to 1 minute of arc. Since this is about the angular resolution limit of the human eye, Brahe's instruments were, despite their relatively small size compared to giant mural quadrants, entirely adequate, and larger size would not have improved their performance significantly (since the eye would have limited accuracy). This suggests that the very large Central Asian quadrants might not have delivered the hoped-for accuracy (but did probably deliver the desired "mine is bigger than yours" prestige).
For analysis and drawings of Tycho Brahe's instruments, see Wesley, Walter G., "The Accuracy of Tycho Brahe's Instruments", Journal for the History of Astronomy 9, p.42, 1978: http://adsabs.harvard.edu/abs/1978JHA.....9...42W or, for a quick overview and pictures, https://www2.hao.ucar.edu/Education/FamousSolarPhysicists/tycho-brahes-observations-instruments
For the instruments of the pre-modern Islamic world, see Stephen P. Blake, Astronomy and Astrology in the Islamic World, Edinburgh University Press, 2016.
I'm not sure why you ask about Newton and calculus, since that is almost entirely irrelevant (the small relevance is that calculus can allow simple understanding of the propagation of uncertainties in data analysis). Is there something in particular you were thinking of?
Do you have any particular measurements in mind? By your choices of scientists, you seem to be referring the positions and trajectories of heavenly bodies. Is that correct?