Great question! I'm writing my thesis on the social life of the Roman Plebs right now, and one of the most important disconnects to get over when you look at ancient Roman life is their different conception of religion. For the Romans, religion was less a discrete aspect of a person's identity and more something that permeated the world. With a few exceptions, Rome didn't go around converting other people to 'their religion,' but adopted new local deities into their pantheon or reinterpreted the local god of war as an aspect of Mars, and so on. Historian Steven G. Wilson says this best, that Roman culture viewed religion as "not merely one component of human experience, but rather as inextricably bound up with all aspects of life."

This different view of religion held by the Romans has given modern historians difficulty. Right now, I'm studying Roman voluntary associations (somewhat like clubs or medieval guilds) called Collegia, formed by the common people in Rome and other cities of the empire. Earlier historians like Theodor Mommsen thought there were three kinds of Collegia: religious clubs that would get together to worship and sacrifice, burial insurance clubs that you'd pay money to and then get a nice funeral, and professional associations (like the Collegium Fabrorum, the builders' group, or bakers', jewelers', or what have you).

However, recent scholars like Wilson and Van Nijf have dug up tombstones and monuments constructed by these groups, and it turns out most were a combination of the three functions. All had some sort of religious function, but a lot of the monuments highlight these people getting together and drinking and having a good time. This archaeological evidence suggests that for average Romans, religion was everywhere and therefore organizations could be nominally affiliated with a particular deity, the wine at your feast could be dedicated to him and if you were in a jam you'd sacrifice an animal to him for good luck, but other than that you pretty much went about your life.

We have lots of evidence for Roman devotion as a society- though some of the stories may be apocryphal, Roman historians wrote about generals who wouldn't go into battle if their ceremonial chickens did the wrong thing, and we've recovered lead 'curse tablets' that people threw into the public baths at Bath, England to bring down the gods' vengeance on their enemies. But as far as I've been able to find, the communal worship of the common people was through festivals (for example, every year there was a neighborhood festival, the Compitalia, to the gods of the crossroads in your neighborhood) and collegia activities.

Although I haven't studied the early Christians, from what I know I'd surmise that their view of religion as an intensely important personal commitment was alien to most Romans (with the possible exception of Vestal Virgins and other members of the elite sacerdotal societies). I believe the letters between provincial governor Pliny and Emperor Trajan record Roman officials being frustrated that the Christians wouldn't just worship the Emperor as an extra god in their "pantheon," which of course from the Christian perspective was impossible.

To the early Christians, then, the average Roman was probably not devout; but to the Romans, Christianity was a kind of devotion wholly outside their experience.