For Byron, rumours of his homosexual adventures had begun to circulate in his last years in England. These became widespread in 1816, spread in part by Byron's ex-lover Caroline Lamb. Together with the accusations from his wife, Annabella Millbanke, regarding martial sodomy and incest, this was enough to shred Byron's reputation. According to Fiona MacCarthy (Byron: Life and Legend) there was at least the perceived threat of "homophobic mob violence" and provided some of the motivation for his exile.

But this was rumour and gossip; they were largely forgotten in time, even if Byron's reputation has never been exactly saintly. His friends and publishers remained extremely protective of his image for over a century after his death. As late as the 1970s, official biographers were not permitted to dwell on or draw attention to the importance of his homosexual or incestuous relationships. It's taken a change in modern attitudes to allow for a freer discussion of Byron's sexuality.