I've read a lot regarding the rotation of troops using maniples. There of lots of conflicting ideas whether they did so using full maniples in the checkerboard formation, using other types of full units, line by line within maniples, or didn't even rotate at all and that relief was the exception. Overall the consensus seems to be unclear.
However, the main thing that is confusing me is how they would have done so during the cohort era, aka after the Marian reforms. We know that the cohort-period legionaries had a system of relief b/c Caesar mentions that troops, who were spread thin along the walls of a camp that was supposed to be manned by more, had to fight continuously without the norm of relief. With the triple line system not existing anymore, how would they rotate? Would they have done it file by file like in HBO's Rome, unit by unit or line by line?
I saw this clip: https://youtu.be/fDTmgHCVWgg?t=20m21s from a documentary in a YouTube video, where they suggest the HBO version as the method. However, I have searched a lot and cannot find what documentary it was from. In addition, the YouTuber mentions that this description was taken from a direct account from Livy. I've spent sometime looking and the only thing I could find of Livy was an account of rotation in Ab Urbe Condita 8.8; however, this was about maniples and not cohorts.
I searched around on a Wikipedia article's talk page to perhaps find other users who were looking for the same thing. In this screenshot: https://imgur.com/a/S2C3R one person mentions that he found a passage regarding rotation of units, but it seemed to refer to the triple line system. My guess is that this is the same 8.8 passage. HOWEver, the response states that they found a different passage about the rotation of cohorts. I cannot find this passage. The wikipedia talk page, if anyone would like to read it: https://en.wikipedia.org/wiki/Talk:Roman_infantry_tactics#Asking_about_individual_soldiers_rotating_the_front_line
So in essence, how did the Roman cohorts rotate troops? A direct sOUrce would be really helpful, thanks.
I can see you've done your homework, which is really good. Essentially, you are correct, Livy 8.8 is the only reference to a rank-by-rank line relief process, and even it is highly unclear and describing a historical period where even if such a process later existed, it could not have existed then. There is, in essence, no good evidence at all for a rank-by-rank line relief process, and as Goldsworthy (1996 208) puts it, this idea was made up by modern historians attempting to explain why the Romans fought in multiple ranks. It did not happen, because there is no evidence it did happen, and it probably would not have been practically possible (ref. Junklemann).
The Romans did rotate individual units, at least they did in the Late Republic with cohorts, and we have explicit descriptions of such a thing, such as the following excerpt from Caesar's description of the Battle of Ilerda, 49 BC:
In this direction our men stood at bay, since, carried forward by their zeal, they had recklessly advanced thus far. The fighting took place in this spot, which was unfavourable both from its confined limits and because p69 they had halted just under the very spurs of the mountain, so that no missile failed to reach them. Nevertheless they strove with valour and endurance and sustained every description of wound. The forces of the foe were increasing and cohorts were continually being sent up to them from the camp through the town so that the unexhausted were always taking the place of the exhausted. Caesar was obliged to adopt the same course of withdrawing the exhausted and sending up supporting cohorts to the same place. (Caes. BCiv. 1.45).
In the above extract, you can see Caesar specifically describes a process in which he withdrew exhausted cohorts, and replaced them with fresh cohorts, and he implies the enemy did the same thing. This probably was only done in lulls during fighting, as it was not practical for units to manoeuvre while the front line was engaged in close combat (Junklemann 241). As for your example regarding defending the walls of camps, troops would have rotated on and off the walls to rest as long as such men were available, but in defending an enormous position such as Alesia, reserves might not be around to swap with the soldiers on the walls (eg. the defence of Cicero's camp in 54 BC, ref. Caes. BGall. 5.45).
It is likely that during lulls in the fighting, when a reasonable gap existed between the foremost soldiers and the enemy, the Roman triplex acies allowed the Romans to bring forward a reserve cohort, or potentially a maniple if the Romans did this sort of thing in the mid-Republic. Note that the Romans actually used cohorts in combat as early as the 3rd century BC, and it is possible that the use of cohorts and this system of rotating cohorts in and out of the front line, but this is all supposition and there is no explicit description of the process.
To answer your question, the Roman cohorts rotated troops by utilising the space available in the lulls in fighting to withdraw a forward unit and put a reserve on in its place. As I understand, most good reconstructions assume that the Romans did not fight with gaps between cohorts but had a solid battleline, so we can probably assume that they didn't do anything overly complex like moving the frontline unit laterally and bringing the rear unit forward, then retiring the forward unit, but beyond that we can only speculate precisely how the process worked.
Sources:
Caesar, Bellum Civile and Bellum Gallicum.
Goldsworthy 1996.
Junklemann 1986, Chapter 5 (this one is in German).
If you can get a hold of an electronic copy, read Sabin's chapter in the CHGRW Vol. I, pp. 399-433.
Not to discourage any further answers but you might like reading this older one while you wait for an answer:
Is the way the Roman army fights in the opening scene of HBO's Rome historically accurate? by /u/XenophontheAthenian and /u/HatMaster12