It seems to me that it would have been more pragmatic had the US government just force the Indians to live together with white settlers, but instead they forced the Indians to vacate their lands and move west. Why was that?
The ideology of Removal was racism masquerading as benevolence. The basic argument went that Native society was too backwards to survive long term exposure to American society from their white neighbors, but Native people should be free to choose whether they continue their barbaric ways on the outer fringe of the civilized world or succumb to the inevitable superiority of white society.
Of course, the real problem for the United States was that their assumed superiority was proving to be less inevitable than they had hoped. Instead of Native societies being absorbed into the American republic, they were busily established their own republics south of the Ohio River (north of the Ohio, that's a different story, which I can get into later but don't really have the time for the moment). The Cherokee Nation was leading the charge on Native republicanism and by the time of the Removal they had penned a constitution, established offices of the Principal Chief, National Council and Supreme Court (modeled after the executive, legislative, and judicial branches of the US government, with a Cherokee twist), and had a national police force, a national newspaper, and higher literacy rates than their US counterparts. For those in American society that sought to groom Native societies into become citizens of the United States (which was the dominant policy prior to Removal), these could be seen as great successes. But the Cherokee weren't adopting an American-style republic as transition toward becoming American citizens. They were doing it as a means of reinforcing their own sovereignty in the face of a changing political landscape.
As the Cherokee were looking more and more like a nation-state in the Western sense, the problem arose that the land the Cherokee Nation was on had largely been promised to Georgia by the federal government in exchange for Georgia's claim on what's now Alabama and Mississippi. As mentioned, initially it was thought that the Cherokee would have become citizens and folded into the rest of the Georgia population, but the Cherokee had no interest in that. They challenged Georgia's right to impose Georgian laws within the borders of the Cherokee Nation in a case that went before the Supreme Court, and they won.
This set the stage for a looming constitutional crisis. The Cherokee Nation was on track to force the United States to recognize its sovereignty. But the Constitution forbids the Federal government from taking a chunk out of existing states without their consent and there was no way that Georgia was going to give up its claim on Cherokee land. Rather than argue for an amendment to the constitution to account for such situations, Jackson argued for ethnic cleansing (using the racist logic mentioned at the beginning of this post). An amendment to address the issue would have been extremely difficult to achieve. After all Euro-Americans would have been the people voting on it, and they were really about to risk having "their" lands taken away by Native nations (and Jackson had already started to argue that if the Cherokee Nation was allowed to split from Georgia, what was to stop the Wabanaki from splitting from Maine - no state was safe!). Strongly "encouraging" the nations of the South to move to Oklahoma was Jackson's preferred solution. Ideally, the Removal policy encouraged westward migration by buying Native lands from those willing to move. But for the most part, the nations weren't eager to sell and the US resorted to illegitimate treaties (Treaty of New Echota for the Cherokee) and the threat or use of force to enact Removal policies.
Long before there was a US government, there was a long period of co-existance with the Indians in Massachusetts. This period came to and end with "King Philip's War". It was this war (followed by many Indian raids during the French and Indian Wars) which soured the attitude of the New England settlers towards co-existence.
There was a long alliance between the Plymouth Plantation and Massachusetts Bay colony settlers and the Wampanoag, Massachusetts and Nauset Indians.
This lasted from the arrival of the Pilgrims in 1621, until 1675, when the Wampanoag leader, Metacomet or "King Philip", launched “King Philip’s War” against the colonists.
So, 54 years, or two generations of peaceful alliance.
The war when it came was perhaps the most dangerous and destructive Indian war ever faced by Anglo American colonists. This war, followed by further Indian conflicts during the French wars, changed colonists attitudes towards the Indians, and not for the better.
When the Pilgrims landed, the local Indians had just been horrifically devastated by an epidemic. The population was reduced from 20,000 Native Americans in Eastern Massachusetts to circa 2,000. This epidemic was probably a European disease spread by survivors from a French shipwreck on Cape Cod.
The Pilgrims landed in a depopulated country. The survivors were weakened, and vulnerable to their traditional enemies further west, who had not been hit by the epidemic.
An alliance was mutually beneficial. The Wampanoag taught the Pilgrims to grow American crops, hunt American game, and thus the first Thanksgiving story came to be.
Charles Mann’s book, “1491”, has a good chapter on the impact of the epidemic and the forging of the alliance. It's a well written book, and interesting, but you need to be aware of some annoying anti-European bias.
Mann generally does a pretty good job of being objective and balanced when he is really engaged with the content. His bias shows when he is skimming through bits.
Here's an example:
After the very good section on the arrival of the Mayflower Pilgrims, how they found New England depopulated by an epidemic, and how the alliance was built, while trying to segue out of the chapter, Mann says that there was peace between the English and the Indians for 50 years before conflict broke out. He then devotes these two sentences to King Philip's War:
"The Europeans won. Historians attribute part of the victory to Indian unwillingness to match the European tactic of massacring whole villages." (page 61)
BANG! There's your bias, just slipping in.
First, we have the false appeal to authority, "Historians". What historians make this attribution? (My guess is none.). (Even if a plural number of historians could be found who said something like this (I doubt it), Mann makes it seem like MOST historians share this view).
Second, we have the dubious and non-factual assertion, "Indians unwilling to match Europeans in massacring whole villages".
The colonists attacked occupied Indian villages three times in the war: The Wampanog town at Mt Hope, Rhode Island, early in the war (in retaliation for the Indian attack on the town of Swansea which was totally destroyed by Indians in the opening action of the war).
The Narragansett fort near South Kingston, Rhode Island, called "The great Fight in the Swamp", where about 300 Narragansett were killed and 70 colonists. The fort was captured, and Indian winter food supplies seized, but most of the Narragansett escaped through the frozen swamp.
The attack on the large fishing camp at Peskeopscut on the Connecticut River, where 150 militia claim to have killed 100-200 Indians with a loss of 40 of their own.
That's three attacks on Indian villages (none of them were "massacres"). Meanwhile, the Indians attacked at least 35 colonial towns, totally destroyed 12 (by burning) and caused 2 more to be abandoned.
The Indians also, never managed to "massacre" an entire town (though they did kill and capture non-combatants and "massacre" some households – as did the colonists).
This was a very hard fought war. The Indians won more engagements than they lost, but eventually, the greater numbers of the colonists proved decisive. It was the colonists ability to outlast the Indians, and absorb more casualties (in the early stages of the war. By the end of the war the Indians had suffered more casualties - perhaps 2,000 - 3,000 Indians (many by disease or starvation as they fled their villages and tried to survive in the woods) versus at least 600 colonists), which proved decisive, NOT, the colonists willingness to massacre whole villages nor the Indians unwillingness to match this.
So, beware of these little bias moments in Mann's book. He is better at objectivity in his main content areas. These biases creep in when he is not paying attention.
The immediate causes of the war are quite complex. There was the death of the previous Wampanoag leader, “King Alexander”, which his brother and successor, “King Philip”, blamed on the colonists.
There was the murder of a Christian Indian, who had been one of King Philip’s advisers (and who had warned the colonists that Philip was plotting war), which the colonists blamed on other advisers of King Philip (instigated, they thought, by Philip himself). There was a trial of three of Philip's advisers for the murder, wherein a jury of 6 Indians and 6 Colonists found the three guilty and hung them. This was the spark that set off the war.
The underlying causes of the war are harder to decipher from the evidence we have, but probably an accumulation of tragic misconceptions and misunderstandings by both the colonists and the Indians, led to increasing Indian frustrations. King Philip also seems to have been a more fiery and intemperate leader than his father and brother had been. The colonists, on the other hand, had probably come to respect the Indians less (and encroach on them more) as they became more self-sufficient and less dependent. You will get more details on the causes and the war itself from these sources than from “1491”.
Jill Lepore, "The name of war, King Philip's War and the Origins of American Identity", 1998
Eric Schultz, Michael Touglas, "King Philip's War, The History and Legacy of Americas Forgotten Conflict," 2000