Talking about pre-colonial kangaroo numbers is difficult due to the lack of extant 'survey' data that exists (no one was really counting/estimating kangaroo numbers until the 1970s, and that only begins to occur on a large-scale due to the ability to utilize 'reliable' aerial surveys). However, this doesn't mean that the question can't be answered, but rather, that the question has to be considered from another viewpoint. For Indigenous Australians, kangaroos (and emus) were the only form of large 'game' that existed on the continent for millennia. They didn't have herdable animals such as cow, sheep, and pig, nor did they have other forms of game animals like deer, bison, or boars. So for many Indigenous tribes, kangaroos filled that niche, and alongside other foodstuffs (native fruits, berries, vegetables, and roots, as well as other animals such as turtles and many types of fish) formed part of their general diet.
The Indigenous Australians had many interesting methods of hunting kangaroos. For the most part, they utilized spears (alongside woomeras) and boomerangs to inflict mortal wounds on kangaroos. Boomerangs being utilized as a tool for hunting appears in rock art throughout the Kimberely region going back tens of thousands of years. More explosively, the woomera allowed Indigenous hunters to throw spears from ~35m, giving them a significant advantage when hunting fast-moving animals such as kangaroos. But in terms of your question, perhaps the most significant method Indigenous Australians utilized for 'managing' kangaroos would be that of fire-stick farming.
In The Biggest Estate on Earth, Bill Gammage essentially argues that Indigenous Australians used fire to both corral animals such as kangaroos during hunts, but also to cultivate the landscape. This cultivating of the landscape (burning down of certain lengths of forests to make way for a more 'park-like landscape' resulted in an abundance of perennial kangaroo grass growing, which Gammage notes was 'very palatable to grazing herbivores. Gammage further goes on to discuss studies which have found that kangaroo populations increase after fire, taking advantage of the terrain and plants which were more suitable to them. To summarise a book which I would recommend for anyone interested in this topic or those adjacent, Indigenous Australians worked to cultivate the land through methods of fire, utilizing it to increase kangaroo populations which could then be hunted as a primary source of food.
To answer your question with this new viewpoint, the Indigenous Australian's didn't require methods of managing 'overpopulation' of kangaroos, and rather, worked to manage their populations to ensure a sufficient level of a primary food source. It was less an issue of overpopulation, and moreso the use of methods to provide for Indigenous tribes (it should also be noted that fire-stick farming had many other uses, including that of removing unwanted trees and plants to make way for fruit/vegetable-bearing ones).
Sources Used:
- Croft, D.B.; Witte, I. The Perils of Being Populous: Control and Conservation of Abundant
Kangaroo Species. Animals 2021, 11, 1753
- Jones, Philip. Boomerang: Behind an Australian Icon. Adelaide: Wakefield Press, 1996.
- Gammage, Bill. The Biggest Estate on Earth. Sydney: Allen & Unwin, 2012.
First peoples in Australia had a much more symbiotic and cultivated relationship with their environment, in that they cultivated the land and assisted production of foodstuffs without damaging the land for future generations (as we know they are an 80,000 year old living culture, sustainability is very important in their lifestyle), that is, producing enough and with a small amount extra fed back into the system so you can return to it months or years later and harvest again. This includes not just fruits and forage, but fish and game. They would hunt and possibly burn down subsections of forest to promote grasslands fit for grazing animals such as kangaroo, but they may also plant additional trees or promote other plants to alter the environment if there was too much present.
European settlers instead took a very western approach by cutting back almost all of the forest they encountered and flattening and planting grazing fields for sheep and cattle, as this can then be used for export and profit - producing more than is required and therefore expanding dramatically. They also forcibly removed the First people inhabiting the lands, killing most or indoctrinating the lighter skinned children.
This had a twofold effect: without First People custodians, the land suffered from a lack of stewardship: things they cultivated became overgrown or died out, and it became far less fertile.
Secondly, the large pastures cut from the forests are the preferred environment of kangaroo and other grazing wildlife; and so the population grew unsustainably as a result. As we continue to expand pasture land rather than reforest, the problem is only set to worsen.