One of my past history textbooks has this quote. "Northern Radicals also had reason to believe that Johnson would deal sternly with the South. When one of them suggested the exile or execution of ten or twelve leading rebels to set an example, Johnson replied, 'How are you going to pick out so small a number? ... Treason is a crime; and crime must be punished.'"
I'm trying to dig and find the origins of that quote, but I'm not making much progress, because I'm just in my undergraduates. I've found many similar quotes. The textbook doesn't give too much context to this. The best contender is this conversation with Senator Benjamin Wade, but its not exact so I might be wrong. https://archive.org/details/lifebenjaminfwa00riddgoog/page/n278/mode/2up
I'm aware about how Johnson had shifted substantially from before his presidency and during, the misperceptions about him, how he was a terrible president, and how the different sides focused on different parts of his life. But I'll stop myself there.
This quote has been bothering me since I can't find it. I'm honestly a bit frustrated/angry at myself for not being able to find it. Most of the references are to the textbook, but the textbook doesn't give a citation.
You're right; it was Benjamin Wade. But it was over the course of at least two conversations and the quotes were ferociously abridged by your textbook authors. See here:
Trefousse, Hans Louis. (1997). Andrew Johnson: A Biography. W. W. Norton. Pp 197-198.