The reason for this question is I’m currently doing some mild world building and am working on the military at the moment.
I’m aware that for something like the phalanx the lines could get fairly deep but I can’t seem to find anything on more flexible systems.
Well for the Romans, the depth of the Goldsworthy has the legions deployed in ranks six deep during the Middle Republic and four deep in the Late Republican armies.
The former probably comes from Polybius' description of the legion where he mentions the maniples of the Triarii were half the strength of the maniples of the Hastati and Principes but with the same frontage. Since the former were twenty files arrayed three deep, that would imply the latter were formed up in the same number of files six men deep.
From the Late Republic period there's mention in Frontinus' Strategems that Pompey at Pharsalus had his legions in ten ranks. But this mention suggests that it was noted because it was not the norm.
For the Imperial period, Josephus in the Jewish War mention soldiers deployed in three ranks a couple of times and on the march six abreast (the implication that a marching column would make a right/left wheel and then turn to face the enemy and thus be arrayed six deep).
Arrian in his Array Against The Alans has his army marching four abreast and arrayed in battle eight deep (suggesting that his marching array was in half files). Furthermore when deployed against the fearsome Alan cavalry, he backed up those eight ranks of heavy infantry with two further ranks of archers.
There's an author who's name escapes me (and is driving me crazy) who suggests that there was no rigid number of ranks that the Romans were always constrained to. That normally a Late Republican or Imperial legion would form up four deep but that local conditions could alter that. An understrength unit could deploy in fewer ranks while a green unit might be formed up deeper. Fighting against hard charging cavalry may need a greater depth than is normal.
Konijnendijk suggests that the untrained and amateur Greek phalanxes weren't always deployed eight deep (as is common wisdom) but that more often weren't. Instead he suggests that the depth varied depending on the number of hoplites present and the amount of space to deploy. Deeper formations could be used when a general wanted to breach his opponents line at a specific location and form up deeper there. Deeper formations added a psychological bonus to the attackers, reassuring them the attack was unlikely to fail (and thus not lose heart or panic and flee) while at the same time frightening the enemy facing them.
So the depth of your battle lines could vary in your world depending on the level of technical skill of the leaders, the flexibility of the military institution, or the type of opponents they would face. Elite units could be deployed in fewer ranks while green units might be arranged in double ranks so that their depth gave the soldiers additional reassurance. They may go in double files if facing heavy cavalry (or particularly fearsome infantry). Also prolonged campaigning will have understrength units that mean either reducing individual unit frontage or fewer ranks.