These changes actually happened well before what is known as the "Byzantine" period and date to Late Antiquity, which can essentially be seen as the period starting basically in the fourth century. The reason for these changes is disputed among historians: One view, that has its origins in Peter Brown, sees these changes as essentially rational responses to changing circumstances. The army of the early Empire was geared towards conquest and offensive action, and the needs of the fourth century military, which were primarily based around border protection from the more aggressive Germanic tribes. However, an opposing view, seen for example in Adrian Goldsworthy's How Rome Fell, is that the late empire essentially could not maintain an army of the sort seen in the early empire. This is largely because the military was far more active, and the frequent civil wars and large scale invasions acted as a meat grinder to the individual soldiers, degrading the quality of training and increasing the reliance on German auxiliaries. The changed equipment towards longer swords and darts can even be seen as a partial "Germanization" of the army. The organization in turn was due to a need of the imperial center to weaken provincial commands due to the greater risk of revolt.

Both perspectives have points in their favor, and I won't give the deceptive cop-out that "the truth is somewhere in the middle". The truth is that we don't precisely know know.