Did the Romans ever tried using their ballistae in a smaller crossbowish way?

Possibly. According to the late-Roman author Vegetius, there was a device called the manuballista, meaning "hand ballista". It was, according to Vegetius, wielded by a unit called tragularii that would fight alongside other skirmishers and light infantry. However, there are reasons to question whether this was really used similarly to a crossbow.

Toward the end of the 1st century AD, the Roman military seemed to be concerned with the limitations of their existing ballistas. If there was a problem then the entire machine had to be disassembled, their range was quite limited, their portability was very poor. An overhaul was needed, and the replacement of artillery equipment was part of a program of military reform in the second century AD that is sometimes called the "Antonine revolution". At the end of the century, under the emperor Nerva, and into the second century AD under Trajan, the artillery was overhauled. Among the new designs was the cheiroballistra, described in a treatise by Heron of Alexandria. The description in Heron and the stuff dug up by archaeologists don't quite match up, but it's close. This suggests that the treatise describes a prototype, as the instructions are a little vague in places and there are clearly some improvements that could have been made, and seem to have been made by the time they actually saw action on the battlefield. A useful part of classical ballista design is that they were highly scalable. The proportions of the parts were dictated not by hard measurements, but by ratios in relation to the size of the shot being fired.

This means that, in theory, handheld versions were possible. However, there was a limit to how small they could be made and still have lethal power. Like, if you make a crossbow too small then you've built a toy that fires toothpicks, not a weapon of war. Ballistas were no different. Before the Antonine reforms, even the scorpio, which was the smallest variety of ballista at the time, needed a crew of two to operate. The new design, in which the arms of the ballista swung inward (a bit like a set of double doors, the technical term for which was euthytone) rather than outward (palintone), was much more powerful but also bulkier and heavier, with a construction entirely of iron. It's questionable how much smaller the manuballista was compared to the scorpio, and they may have been functionally the same. Vegetius doesn't really help us out here, he makes the vague statement

"what they now call manuballistas they used to call scorpions, naming them in this way because they inflict death with small, slender darts"

The soldiers may have initially thought of the manuballista as just the new version of the scorpio, but Vegetius tells us that this was because of the small size of the darts, not the size of the machine. But archaeological evidence helps us out here. Even the smallest manuballista components discovered suggest a machine that may have been too large or front heavy for one person to hold, but it depends on how one interprets the evidence to reconstruct what the machine was like.

So they probably couldn't use it like your average crossbow. That is to say, it wasn't as simple as pulling back the string, loading a bolt, pointing it at the enemy with one's hands, and activating the release mechanism. If these were used by one person, then it may have come with a stand, perhaps similarly to early firearms like the handgonne, which had a similar front heavy design but was also small enough to be held by one person. They may also have knelt when firing and balanced it on their knee. Or perhaps it wasn't quite as front heavy as is currently thought and it was fine (though if that was the case, then one would also have to wonder what made the crossbow sufficiently better to eventually supersede it). This is just speculation based on the problems of the design, however. In terms of their functionality, they were basically the same in that they fired a small bolt with terrifying power, but a manuballista was certainly more complicated and unwieldy than a crossbow and could not quite be used in the same way because of that.