The "Subarmalis", "Thorocomachus," "Himation" or "Peristithridion" is often mentioned in both Roman sources and Roman art. For example, the tombstone of Severus Acceptus appears to show a qulited gambeson, and so do the Split and Eleutherna Sarcophagi.

Tombstone of Titus Calidius Severus: https://i.pinimg.com/236x/37/2e/2d/372e2d80df393b8c775ef9e919114443--roman-sculpture-roman-art.jpg

Tombstone of Severus Acceptus: http://www.livius.org/pictures/turkey/kadikoy-chalcedon/tombstone-of-severius-acceptus/

Furthermore, a quilted linen Greave liner has been found at Vindolanda and a second from Dura Europos. A leather fragment from Ballana in South Egypt might be from such a gambeson called a Peristithridion.

Fabric Greave liner: https://i.pinimg.com/736x/a6/14/8d/a6148dbfcb954a354799c3c28c521ad2--warfare-armour.jpg

The question is "did they always wear it?" And the answer to that seems to be "no." Segmentata technically doesn't need it, other than to make it more comfortable - a properly made segmentata (Say by Matt Lukes) can take a hell of a beating without a single dent. Scale and Lamellar are also much better at absorbing shock than Maille as well. Chainmaille is what technically needs it, because of its flexibility, but even then we don't usually see it in art alongside the "Subarmalis"

It also seems to have been used as armor on its own (particularly during the Byzantine Era when it's called a "Kavadion" which is what turns into the Gambeson proper of the High and Late Medieval Era). Even at its height, the Roman Empire couldn't supply all of its troops with armor. In the late Empire we know they could supply about 50% within each unit with metal armor (still 10 times higher than anyone else). In the early empire that ratio may have been higher but we don't know. Quilted linen provides excellent armor for low cost for those who couldn't afford superior bronze or steel armor, and had been in use for centuries. As such, it was probably often used by those who did not have armor.

Sources:

• Graham Sumner, Roman Military Dress, 170-175

• M.C. Bishop, Roman Military Equipment, 63, 98, 263