Other bureaucracies are complex too, indeed one would struggle to find a bureaucratic state that _wasn't_ seen as complex. Byzantium did have a complex political system, with aristocracy and bureaucracy, imperial succession battles that look a little different, but one can't really say that they're uniquely complex. That Byzantium gets labeled this way is a reflection of who's describing them- its not they themselves, obviously. It also reflects the difficult for a modern scholar in studying Byzantium-- to do justice to the subject you need a lot of languages, the price of entry to Byzantine studies is a lot of work learning to read things that most people can't

So the characterization as "Byzantine" comes from Western Christians looking at Orthodox Byzantium, and doing so rather recently-- this is an "Orientalist" epithet, and becomes popular in the 19th century. While contemporary politics of the 19th century saw the Ottoman Empire as the "sick man of Europe", historians of the time located that sickness all the way back to Byzantium. One particular quote is sufficiently useful as a resume of 19th century complaints that every volume rolls it out, no reason for me to omit it, because it's splenetic and concise:

Of that Byzantine empire, the universal verdict of history is that it constitutes, without a single exception, the most thoroughly base and despicable form that civilization has yet assumed. There has been no other enduring civilization so absolutely destitute of all forms and elements of greatness, and none to which the epithet "mean" may be so emphatically applied ... The history of the empire is a monotonous story of the intrigues of priests, eunuchs, and women, of poisonings, of conspiracies, of uniform ingratitude.

That's from A history of European Morals from Augustus to Charlemagne [Lecky: 1869]. Not well known today, William Lecky was a then renowned Whig historian and political moralist during Victoria's reign, someone who saw the world moving from less advanced to more advanced systems, and he located in Byzantium a great deal of what he considered "less advanced". He finds in Byzantium all the complaints that a Protestant might have about Catholicism, magnified by misogyny and Orientalism. "Priests, eunuchs and women" -- an trinity of effeminate intrigue, repellent to Lecky, a modern Unionist Liberal.

You can see in Lecky's denunciation a concise list of the reasons an 19th century British Liberal might see Byzantium as a poor example, most of which reflect more on prejudices of the day than they do on Byzantium itself.