Specifically the period between the reign of Diocletian and the fall of the Western Roman Empire.
Basically you can summarize the changing Roman army of the late empire with 4 points. These are as follows. 1: size of the army, 2: specialization of the army, 3: barbarization of the army, 4: restructuring of the Roman army.
1: It seems that the size of Roman units and the army in general was smaller. To some degree this can be attributed to the difficulty Rome had with recruiting and maintaining a large standing force as had been seen during the principate. This is largely due to the prevalence of civil wars which caused economic disruption and also wasted Roman strength fighting itself. There wasn't as much of a need for a few huge armies anyways. Instead, many smaller armies were better suited to border patrol duties along the frontier. Remember that largely it was a matter of maintaining the empire rather than forging one at this point. Creating many smaller units also opened up new positions which the emperor could fill at his discretion to gain political support. Smaller armies were also less of a threat if they turned against the emperor. During this period internal threats were often the emperor’s greatest concern. Once again we see civil wars having a huge impact on the late empire.
3: Rome had a harder time recruiting troops due to the economic and military disruption of civil wars. There were also issues with supply, army compensation, and recruitment which meant putting together an army was harder. Therefore they often turned to new recruits drawn from immigrating barbarians. Rome had more routinely been settling peoples from beyond the frontier to become colonists in buffer regions along the border. In exchange for settling, these tribes often had to give a set number of troops for a period of time. These were snapped up by the recruit starved army and trained/drilled/equiped like normal soldiers. Over time the officers would also become barbarians but these were most likely heavily romanized. The barbarians brought with them a preference for different types of equipment (trousers, round shields, spears) and battle behavior (barritus, shield wall). Claims that this led to decreased performance of the Roman army is largely discredited by historians.
4: All the changes above would slowly and naturally lead to a reformation of the army. A more abrupt and directed change came with the rule of Diocletian and Constantine. Diocletian's reforms are credited with splitting the Roman army into border and interior forces meant to better protect the empire. Border forces, known as limitanei, were smaller, less well paid, and did not have as high physical requirements. Limitanei were distributed along the frontier in permanent forts. They were tasked with patrolling and scouting duties both on land and along rivers. In this capacity they relied on more mobile troops, dealing in stealth and speed. Enemy raiding parties of up to maybe 500 could be stopped. Anything larger and the border forces would retreat to their forts to await the field armies.
The field armies were known as comitatenses. One source says there were 7 of these armies in the west and 5 in the east. Such forces had the heavy infantry we expect from the Romans including elite palatina troops and also new heavy cavalry and more auxiliary troops. In contrast to earlier legions who were stationed in dedicated camps, the comitatenses were billeted throughout cities to reduce the strain of their upkeep. The field armies were meant to be shifted around to respond to penetrations of the border, perform large combined campaigns, or even defend against internal threats of usurpation.
Overall we shouldn't think that the new Roman army necessarily came about because the old legions were outdated. This change was more a product of its troubled times . The old army simply could no longer be supported. Armies of the late empire were arguably not as powerful as they once had been but still proved superior to their enemies.
As for details of the army's performance we don't have as many sources to go off of. Rome still won many of its battles but was more fragile and had to be used with a bit more care and finesse than before.
Hope this quick overview helps. I'd be happy to answer more questions or would refer you to the book "The Complete Roman Army" by Adrian Goldsworthy