The example that was given was that they'd pay their soldiers based on the number of heads they brought back on their belts. What was this dynasty? And why did it fail, if its system was so successful?
This sounds most likely the Qin Dynasty, which was based off of the Legalist philosophy which emphasized rule by the state through a system of harsh punishments and generous rewards.
As for why the system fail, well military efficiency only gains you so much when you apply the same system to managing civilian affairs. Upon the death of Qin Shi Huang, his entire empire started fragmenting and rebelling against the harsh authoritarian state he had imposed.
There's actually an old chinese axiom that is used to illustrate this divide, between the Qin Dynasty and its Han Dynasty successors.
An empire won on horseback can not be ruled from it.
Meaning the military means necessary to conquer, will be insufficient for maintaining those same conquests.
This sounds like the Mongols, although without any more information I can't say for certain. They were supposed to have employed a similar system during their invasion of Central Asia. Can you give me more?