This is not my field of expertise, but I think I've read enough on the topic to attempt an answer. Somebody more knowledgeable can feel free to correct me if I overlook anything.
There were many single points in time where the Indian Removal Act could have been prevented. The act was very controversial, with many Christian missionaries, lead by Jeremiah Evarts, protesting the act. According to historian John Andrew,
Evarts' tactics were clear. He planned to organize a phalanx of friendly congressmen to present the case against removal on the floor of the House and Senate, hoping to convince enough Jacksonians that the immorality of removal required them to vote against the Indian Removal Bill. At the same time, he would continue to barrage the public with letters, pamphlets, and articles on the Indian question, along with whatever other information might create a groundswell of public opinion against removal.^1
In Congress, New Jersey Senator Theodore Frelinghuysen made a six hour speech stretched over three days.^2 The Senate passed the Indian Removal Act by a vote of 28 to 19 on 24 April 1830. Tennessee Representative Davy Crockett also argued against the bill (even though it later cost him his seat), and the bill barely passed the House by a vote of 101 to 97 on 26 May.
It's been a few years since I've read the book and I unfortunately don't have a copy, but from what I remember H.W. Brand's Andrew Jackson: His Life and Times Jackson truly believed that the Indian Removal Act was inevitable^3 and probably would not have had any reservations when he signed it into law on 28 May.
The problem with trying to pinpoint a single point in time that something could have been changed is that you have entire systems at play, and just one small change could have prevented the Act, but then again, it could have not. You can never really know because it was counterfactual.
Sources:
John A. Andrew III, From Revivals to Removal: Jeremiah Evarts, the Cherokee Nation, and the Search for the Soul of America (Athens: University of Georgia Press, 1992), 220.
Daniel Walker Howe, What Hath God Wrought: The Transformation of America, 1815–1848 (Oxford: Oxford University Press, 2007), 348-352.