(Copy-pasted answer from another recent question; it's coming up a lot lately...)

If the rumors are true at all (they weren't reported as fact even in ancient sources), he sang:

Yet his measures, popular as their character might be, failed of their effect; for the report had spread that, at the very moment when Rome was aflame, he had mounted his private stage, and typifying the ills of the present by the calamities of the past, had sung the destruction of Troy. (Tacitus, Annals, XV, 39, tr. J. Jackson)

If an instrument were involved, it would have been a cithara, a fancy kind of lyre, typically used by singers to self-accompany. Nero was reported to play a variety of instruments -- including the water-organ, aulos, flute, and bagpipe -- but his main instrument seemed to be the cithara. (It's variously translated into English as "lyre", "harp", or even "lute", but the Latin of these specifically mentions the instrument by name.)

Having gained some knowledge of music in addition to the rest of his early education, as soon as he became emperor he sent for Terpnus, the greatest master of the lyre in those days, and after listening to him sing after dinner for many successive days until late at night, he little by little began to practise himself, neglecting none of the exercises which artists of that kind are in the habit of following, to preserve or strengthen their voices. (Suetonius, The Life of Nero, 20, tr. J. C. Rolfe)
Last of all to tread the stage was the sovereign himself, scrupulously testing his lyre and striking a few preliminary notes to the trainers at his side. (Tacitus, Annals, XIV, 15, tr. J. Jackson)
He began by reciting a poem on the stage: then, as the crowd clamoured for him to "display all his accomplishments" (the exact phrase used), he entered the theatre, observing the full rules of the harp — not to sit down when weary, not to wipe away the sweat except with the robe he was wearing, to permit no discharge from the mouth or nostrils to be visible. (Tacitus, Annals, XVI, 4, tr. J. Jackson)

Lyres/citharas are an instrument somewhat like harps -- a layperson seeing the instrument would probably happily call it a "harp" -- but with its arms and strings parallel to the soundboard rather than perpendicular. A cithara was kind of lyre for professional musicians, with more strings. (Note that, while "cithara" is the etymological root of "guitar", the instruments are unrelated except in the sense that they're both string instruments. The Romans did have lute-like instruments, but they didn't have the popularity of lyre-like instruments.)

Could a modern musician play a cithara, in the sense that they could coax a tune out of it? Yes, absolutely. Really, there are only so many different kinds of objects with well-behaved vibrational modes (strings, tubes, taut skins, etc.), and only so many ways to amplify those vibrations acoustically, that it's often pretty obvious if you have an example of an instrument (and/or a good rendering of it), how you would have made it make sound.

The problem is more about tuning and other aspects of musical performance -- not just sounding, but sounding good to the audience. For example, the musical notes (i.e. what frequencies an instrument can play, out of infinite possible frequencies) that Nero would have been playing is a matter of conjecture. The Romans wrote a lot about social aspects of music but not (to my knowledge) detailed guides to tuning or (more generally) about their theory of music; it's usually assumed they are using Greek music theory with little modification. (Nero is, after all, playing a Greek instrument, sometimes even in Greek music competitions). I wrote a little bit about the tuning problem in an answer to this question.

That particular problem is more of a problem for stringed instruments, though. For an instrument like the Roman tuba (a long bronze trumpet-like instrument), there's no question of what notes it could play. Roman horns didn't have valves like modern brass instruments, so you could only play the harmonic series -- think of fanfares, or playing "Taps" on a bugle. It's less a matter of conjecture what tuba music would have sounded like because there's just fewer degrees of freedom there.

Or take the water organ mentioned above. The hydraulis was an early pipe organ that used an ingenious hydraulic mechanism to maintain a constant air pressure. Fixed-reed organ pipes play notes that depend on their lengths, so we would know what notes it would play just from measuring them. I'll leave you with one more example of Nero neglecting the pressing issue of the day (a Gallic uprising) in favor of his musical obsessions:

Not even on his arrival did he personally address the senate or people, but called some of the leading men to his house and after a hasty consultation spent the rest of the day in exhibiting some water-organs of a new and hitherto unknown form, explaining their several features and lecturing on the theory and complexity of each of them; and he even declared that he would presently produce them all in the theatre "with the kind permission of Vindex." (Suetonius, The Life of Nero, 41, tr. J. C. Rolfe)