Pretty much all said in the title, but were there any colonists who thought what they were doing was bad and if so, do they have written accounts (I.e. diaries, journals, etc.)?
Portuguese priest António Vieira, alongside Bartolomé de las Casas, strike me as partial examples of that.
Vieira’s work is extensively studied in Portugal/Brazil (it’s mandatory reading on high school curricula, at least in Portugal) and he’s considered the father of Brazilian literature and, to some extent, one of the precursors to the formation of Brazil’s national identity.
His “Sermon of Saint Anthony to the Fish”, written in Northern Brazil in the second half of the 17th century, is a critique of the Portuguese enslavement of indigenous tribes. Vieira was ultimately involved in a political push in Lisbon (that he himself led) to have the King extend “basic rights” as “Christians” to the native peoples of Brazil.
A good overview of the Jesuits’ outlook on native Brazilians can be read here, including some clarifications on Vieira’s political stance:
McGinness, Anne B. (2018). "The Historiography of the Jesuits in Brazil Prior to the Suppression".
https://referenceworks.brillonline.com/entries/jesuit-historiography-online/*-COM_209645
I haven’t read this since college, so I cannot speak to it in any detail, but you might want to start with Bartolomé de las Casas’ A Short Account of the Destruction of the Indies. It’s a first hand account—and he was an influential figure, an early advocate for human rights, an opponent of slavery, and—I think—colonialism to some extent.
Edit: bad sentence made slightly less bad