Hadrian had absolutely no qualms in crushing his opponents in the process of becoming emperor after the death of Trajan (the adoption was believed to be irregular for a number of reasons, but it was a moot point since he had the approval of the Senate and the army). He had Lusius Quietus and three other prominent senators killed without trial by his right hand man Attianus just because they were close to the former emperor. Later he stripped Attianus of his power (and he probably had him executed) because he didn't like his ambition.

He had a very tense relationship with the Senate and ruled mostly by decree, even though at this time the approval of the Senate was just a pro forma. So bad was his standing with them that the Senate wouldn't accept to deify his after his death; Antoninus had to stamp his foot really hard to convince them to do it.

Fronto later recalled (in a letter to Marcus Aurelius) that he feared rather than loved Hadrian.

The Jews hated him so much because of military and religious repression that every time he is mentioned in later rabbinical texts they curse him with the expression may his bones be crushed - even though this isn't very historically relevant, he was seen as a tyrant and an antisemite by them.

He was certainly a philantropist and a big spender in public projects, but he was no soft literatus.

I dunno much about Marcus Aurelius, but I can answer on Hadrian. Among the things of which Hadrian was accused by anecdote or as a matter of historical record:

1. Driving a pen into the eye of a slave.
2. Presiding over a massive genocide in Judaea.
3. Participating in massive genocide in Dacia (during Trajan's principate.)
4. Executing former consuls and senators and some of his ambitious allies at the outset of his reign.
5. Executing and imprisoning senators at the end of his reign.
6. Executing an architect who made fun of him.
7. Possibly executing various figures during the middle of his reign.
8. Possibly murdering his lover so he could use the death for some magical purpose.
9. Killing his brother-in-law and his brother-in-law's grandson near the end of his reign (they may have had it coming.)
10. A reputation for narcissism—for always believing he was the best at all things, and for tolerating no disagreement on the matter.

And so on. One thing his successor is said to have done is release many of the men Hadrian, in his paranoia, had imprisoned and marked for death. There is a case to be made that Hadrian's personal secretary, the historian Suetonius, wrote his history of Domitian as a critique of Hadrian—and a case that Hadrian's rule bore similarities to those of Nero and Domitian. If you love historical drama, and want a second look at Hadrian, read Cassius Dio or the Historia Augusta. It's terrific stuff.

Trajan was not particularly a saint either—rather, he was successful (except those latter wars, which plunged the empire into crisis, and which cast a bit of a pall over the start of Hadrian's reign on account of Hadrian having to "withdraw" from territory the empire had de facto lost already)—but that aforementioned genocide in Dacia was his doing. The lectures of the German historian Theodor Mommsen (who taught in the 1800s) asserted that the depopulation was so total that the language and culture of the region were lost.

This is not to say they were particularly worse than other emperors, but rather that their benevolent reputations in popular history is a little overstated.

Edited to add: some historians (including Hadrian himself) argued that the stuff in the above list didn't happen, or didn't happen the way stated here. There's something to that—the Apollodorus stuff is fun to run down—but also a lot to the story that certain genocides happened at certain times, and that certain figures vanished from the historical record at certain times, and in certain ways that cast doubt upon Hadrian's own account of what happened.

Bonus question: I once read about Marc Aurel consuming opium, is this true?