Was there a primitive version of modern fingernail clippers? Did they use a file? Just let them grow and then break off at some point?
The daily activities of prehistoric man likely determined their length. Once you move into recorded history, we have records of stones being used, then paring knives (the nail carved or "peeled" rather than cut), then scissors, then the modern clipper.