I am not an expert on this but I'll give it a go!

Unsurprisingly it depends not only on the time but also the place we are talking about, e.g. a soldier in a relatively peaceful province would have a different experience to someone stationed in uncharted territory like Wales or Northern England, but I'll try to outline some basic characteristics.

If I'm not mistaken legionaries were not officially allowed to marry or have families, and would spend most of their time as a relatively close-knit group (speaking a common dialect, the servo militaris or 'camp Latin') and not mingling much with the indigenous population of the province they were stationed in. In Roman Britain for example, the military was pretty much sealed off within a vast military zone governed by martial law that civilians generally weren't allowed to enter from outside. Within this zone there were a few smaller towns (vici) that were connected by a road network, and soldiers could be allowed leave (commeatus) to go there for trade or recreation, but that's pretty much it. (Similar to the area around the Wall in Game of Thrones, if that means anything to you.) It is important to remember that even in times of peace, Roman soldiers in the provinces were still occupying armies and often a tool of state control. They played an important role in administering the provinces throughout the Empire's existence because, crucially, they were a largely literate and already well-organized body.

Religious worship was an important part of military life and soldiers had their own religious culture, clearly favouring specific deities over others. Mithraism is probably the best example for this; being a purely male oriental mystery cult made it a perfect match for tight-knit military communities in far away lands, and outside of Rome it was basically a purely military phenomenon. Thus worship and building shrines would fill some of a soldier's free time.

Building is a good keyword here: much of the Pax Romana was achieved by a strategy of pushing forward into foreign territory and consolidating gains by erecting garrisons (and also building forts over capacity as part of a 'smoke and mirrors' strategy), so legionaries spent a lot - and seriously, I mean A LOT of time building and maintaining forts and camps, day in and day out over weeks and months. (Edit: For example, it is estimated that constructing the fortress at Inchtuthil in Scotland alone required a total of 16.5 million man-hours) Within garrisons, soldiers also had specialised duty assignments such as working in workshops, building bath-houses, working at the hospital, working clay, etc.; because of this garrisons were surprisingly self-sufficient in some respects.

We also know from the Vindolanda tablets - probably the single best source of military everyday life as seen 'from below' - that soldiers would maintain personal friendships: gambling together or sending letters to fellow soldiers who had been transferred to other posts. Soldiers also organized in patronage networks or social 'clubs' (collegia).

Finally, Roman legions were regularly rotated between provinces and thus soldiers would spend a considerable amount of time travelling.

I am sure there are many other factors but I think the ones I mentioned above - religious worship, administration, construction, duty assignments, interpersonal relationships, travelling, sometimes recreation - would fill a fair share of the average soldier's time while not engaged in actual fighting. Most of the information I used comes straight from David Mattingly's Imperial Possession, which draws together a wealth of information on life in Roman Britain from an everyday life perspective! I would highly recommend it.

Edited for spelling and some additional information.