Did fingernails and hair stay naturally limited somehow?
Cutting tools are far older than modern humans. Our species - Homo sapiens is only about 200,000 years old, but the Acheulian hand axe is 1.6 million years old. Homo erectus, Homo antecessor, Homo heidelbergensis and Homo rhodesiensis were all using it before us.
Contrary to what some people may think, stone tools can be very sharp. Obsidian edges are thinner and cleaner than steel, and some surgeons prefer to use them rather than steel scalpels. Especially cosmetic surgeons, who want to leave the smallest scar possible. Even stone edges made from common chert are quite capable of cutting hair or nails.
Of course, just because they could cut them doesn't necessarily mean they did. Personal grooming habits of people who lived tens or thousands of years ago are unknown to us, because we only have bones to go by.
Nails will wear down naturally if you do a lot of labor by hand. They are like animal claws, which remain trimmed through normal use. Only pets who sit around in homes instead of surviving in the wild need occasional trimming.
Hair can be plaited to keep them tidy, or tied out of the way with string or rawhide or sinew, which even our distant ancestors had. So it's possible some of them kept their hair long. We really don't know, but we do know they had tools that could have cut hair over a million years ago.