No, for most of the empire the nerds held the political power.

Success in Roman politics depended upon education and connections. You had to know the right people, and you met them while attending elite schools that trained you in the rhetorical skills, philosophical knowledge, and deportment necessary to move in elite circles in the Greco-Roman world. This education - paideia, in Greek - started with memorization of Homer (or Vergil) and, if completed, was basically a graduate degree in literature and/or philosophy. It required significant expense and years of training to complete. But without it, it was hard to be taken seriously by the ruling elites.

By the third century, the army was playing a central role in selecting the emperor (in many cases, by deposing the sitting emperor), and it was possible for an uneducated military general to make his way to the top. But these new men struggled to win acceptance from the established senatorial aristocracy, and usually didn't live long before being assassinated and replaced. Education gave you connections and taught you how to get along with the rich and powerful; without it, it was difficult to maintain a powerful position in the heartland of the empire.

Over time, this created something of a cultural divide between educated urban elites and the military families which rose to prominence on the empire's frontiers. Philip von Rummel has argued that many of the fifth-century texts complaining about 'barbarians' are actually talking about these Roman military men who gained power in the army on the frontiers, and whose lack of education earned them harsh criticism from the old, wealthy elites in Rome and other urban centers of power.

It was only with the division of the western empire into smaller kingdoms ruled by these military generals (in the fifth century) that the 'nerds' finally lost power to the less educated military strongmen.

If you want to read more, I'd recommend P. Brown, Power and Persuasion - a short and well-written book on the importance of education in Roman politics and power.