St Andrews was unusual in that between 1552 and 1805 they used rabbits. There was a financial incentive for this, because rabbiting was a huge industry when every part of a rabbit had a use and there was always an increasing supply. This benefitted both the landlords who sold on the rabbits and the commoners who caught them - both received financial rewards - but the golf course would have held little revenue for commoners until tourism as an industry developed as we know it today.

Mass killing the rabbits was legalised in 1805, as the tourist golfer population and their interest in st andrews was rising. Golf itself was being standardised, and rabbits left too many of their own holes in the course to disturb matches. There was a 16 year struggle between the two sides until 1821 when the foundations for an established course were laid out.

Because the golf course is known to be over 500 years old, but rabbits were permitted in 1552, I would assume that rabbits were already being used and permission was granted as it was a difficult practice to curb (as is often the way) but it would otherwise have been livestock, probably sheep. Modern grass maintenance arrived in the mid-victorian era when the basic lawnmower was invented, in 1830, so I would again assumed livestock were used until later in the 18th century when St Andrews would have been able to purchase one.

I was able to find this internet history for quick reading, but the 'definitive history' recommended is St Andrews, The Evolution of the Old Course: The Impact on Golf of Time, Tradition and Technology by Scott MacPherson. Hope this was helpful.