For us today, religion is a matter of conscience, but in the Roman empire religion was public and political. The official sacrifices to Jupter and the emperor were meant to ensure the health of the empire, and citizens participated as an act of civic duty rather than from personal conviction, deeply held beliefs, or 'faith'. By the third century, as internal political weaknesses threatened the empire's existance, various emperors made certain sacrifices manditory. This wasn't because they wanted to prevent their subjects worshiping other gods (Roman state religion wasn't exclusive). But they did want citizens to prove their support for the emperor, and their dedication to the empire, by invoking divine support for the state in the appropriate and traditional ways. There are a few well-known cases of Christians refusing to make these sacrifices, but a recent book has convincingly argued that most were fine with it - state religion was civic duty, and you could do your part and make a sacrifice on the emperor's altar without denying your personal beliefs (Rebillard 2012).

At the end of the third century, Diocletian won a civil war against his rivals and attempted to secure his position. Many (most) of his predecessors had been assassinated during the previous 50 years, and he needed to establish his control firmly and permanently. One of the many stabilizing reforms he instituted was to intensify 'traditional' Roman religion, requring greater commitment from all his subjects to the worship of the old gods in what he considered to be the appropriate, traditional ways. This resulted in one of the first truly empire-wide persecutions of non-conformists like Christians, because Diocletian considered failure to worship Rome's civic gods as hostility toward the empire's stability.

Diocletian's successor, Constantine, had to defeat several rivals before he could take control of the whole empire, and he followed Diocletian's lead in emphasizing the importance of religion for stabilizing his new power. Unlike Diocletian, he chose to break with tradition and invested in the increasingly popular new religion that had survived Diocletian's persecutions. Christianity became established as the new official faith whose proper observance would ensure the health of the empire. In making Christianity official, Constantine didn't fully shut down traditional Roman observances, but he did invest heavily in churches for the new imperial faith, and he organized the Council of Nicaea to force Christians to figure out the 'official' version of their religion (much like Diocletian had defined the correct way to worship the old gods). (See Van Dam 2009 for a good discussion of Constantine, Diocletian, and the establishment of Christianity.)

So by the end of the fourth century, emperors had - with the exception of Julian, who had tried to revive state support for paganism - relied on Christianity to provide the divine support needed to ensure the health and stability of the empire. The old civic religion wasn't fully suppressed, but it was becoming unpopular, and your best road to success in public life was through observing Christian rituals.

Between 389-91, emperor Theodosius I made the old civic religion officially illegal. Specifically, he was targeting the sacrifices and festivals that Christianity replaced: the official, public observances that had been meant to keep the gods on the side of the emperor. By making these illegal, Theodosius was finishing what Constantine had begun: shifting the empire from the old gods onto the new one. These laws weren't really targeting individuals' beliefs; we know of high-profile pagans who continued to hold their old rituals after Theodosius' edict. They were shutting down the old civic faith, and ensuring that the empire's spiritual health was tied only to the Christian god.

There were some persecutions of pagans after this. The most famous is Hypatia, a pagan philosopher and teacher (and a woman) who was murdered by a mob in Alexandria. But mostly, Christians seem to have assumed that everyone was Christian now. They talked a lot about Christian triumph, and pulled down some of the old temples, but there's little firm evidence for pagan witch hunts (A. Cameron 1994). Paganism had been a state cult, and without state support, 'pagans' were basically just bad Christians in need of correction (P. Brown 2004).

So to answer your question: the Roman empire never really cared what you believed (well, Justinan may have, but he doesn't come around until the sixth century). But Romans did always care that they had the support of the gods backing up their imperial power. That meant there was always a close connection between state power and civic religion. For most of the empire's history, that religion was public sacrifices to the emperor and the traditional Roman gods. After Constantine, it was Christianity. But the underlying logic - that properly worshiping the gods (or God) ensured the empire's health - remained true throughout the empire's history.