This one is pretty straightforward. Crockett was a Whig, and the Whigs were against it. He doesn't seem like he would have been a Whig. Outwardly he seemed the epitome of a Jacksonian Democrat, being a self-made man of humble origins from the new western states ( even Jackson's home state of Tennessee). This he found useful, as the Whigs made much of him for this reason. Like parading around with log cabins in election rallies, it helped them to hide their party's primary allegiance to wealthy, mostly eastern elites.

It turns out that party affiliation was a far more important factor in the votes than the number of resident Indians in any state: Fred S Rolater found 92% voted with their party. Crockett was also not the only southerner: 13 National Republicans ( Whigs) would also vote no, including one who represented Salem, NC, which had a long Moravian tradition of missions to the Indians. Crockett attributed his failure to be re-elected in 1831 to this vote. However, he was able to get elected back to Congress the following term. His failure in the 1835 election has been attributed to his taking too much time from politics, in order to promote his autobiography.

Rolater, Fred S. "The American Indian and the Origin of the Second American Party System." The Wisconsin Magazine of History 76, no. 3 (1993): 180-203. http://www.jstor.org/stable/4636431.