There are several ways to look at your question.

It seems, though, that rather than speaking of professional standing armies in and of themselves, you're more curious about the tactics and such.

For that, first, you should look at the Battle of Adrianople in 378. In a battle between a Roman army led by the Emperor Valens himself, and a coalition of various Goths, Avars, and Huns, the Romans suffered one of the worst defeats in their history at the hands of the Goths, largely due to a Gothic cavalry charge crushing the left Roman flank. (Maurice's Strategikon, George T. Dennis translation)

After the battle, the Romans began integrating the "tribal" (used here to describe the various Germanic, Hunnic, and Avar groups) warriors into their armies and adjusting their tactics.

By the time the late 6th century rolls around, the Roman army itself had undergone massive changes, thanks to changes in technology and the adaptation of tactics from the Goths, Huns and Avars.. According to Maurice's Strategikon (a Roman military strategy handbook, written by the Emperor Maurice), among the most important skills for Roman soldiers was the ability to fire arrows "...rapidly mounted on his horse at a run, to the front, the rear, the right, the left."

As to why cavalry became more important, there are a few theories. One in particular was pointed out in the introduction to the Strategikon by George T. Dennis, the translator. According to him, the stirrup had begun to come into widespread use, allowing a mounted lancer greater stability in the saddle, this providing more force for their thrusts. This, along with the invention of the high-backed saddle, allowed the early cavalry of late antiquity to evolve into the medieval knight.

If I've overlooked anything, please let me know! I'm always looking for an opportunity to learn.