Short answer: racism.

The governments and people of the US, particularly in the west, always wanted to expand westward into every available possible piece of land. Native Americans were seen as existing outside of the white institutions of law, agriculture, and Christianity.

The fact that the tribes in question had made significant efforts at adopting those institutions is why it took longer for them to be removed and why it was very controversial.

Andrew Jackson and people like him deeply hated Native Americans and viewed their presence as a threat.