This question comes up from time to time and has a section in the faq.
It’s also a big question, for which I'll highlight a few important points focusing more on Spanish America which I'm more familiar with including a summary below (updating this from various earlier answers) – hopefully others can add more on British America.
Importantly, looking at demographics, a larger part of the pre-hispanic native population was concentrated in Central and South America in comparison to North America. This was influenced but not "reversed" by the demographic catastrophes that struck native people in Mexico and other places post-conquest. The two largest and most populous empires in the Americas at the time of European contact, the Aztecs and Incas, where in central and south America respectively. The Spanish would build on their structures and they would form their colonial centres, where Spanish control was most secure by the late 16th c. Of course there were also important states and highly developed cultures in North America – this is just to give a rough comparative idea of demographic density in both regions.
Economy & justifications for colonial expansion
In Spanish America, indigenous labor was at first crucial: the early conquistadors went in search of gold, and with the discovery of silver in Potosí silver became the main resource holding together the Spanish empire. The conquistadors’ other main incentive was to set up their own encomiendas, which overall not many of them achieved. The encomienda was the right for the labor of specific indigenous communities - and not for their land per se. While the encomienda system eventually declined, this focus on labor in order to extract silver and other resources to send to Europe continued. With the huge demographic crisis of native people in many parts of Spanish America, African slaves were increasingly brought in for work (which is another story).
For the English, at least early on, the focus was not so much on indigenous labor but more on the right over American land. John Locke famously made the argument that there was a right to take over "vacant land" not held by other people. This argument was so broad that it was regularly applied to land were Native Americans lived, who were not Christians and so did not exactly count as people at that time. In the British American case, this was also out of necessity: as mentioned, the pre-columbian population density of people was simply much lower in the modern-day U.S. than e.g. in the regions of the former Aztec (central Mexico) and Incan (Peru) empires. There were also no major silver or gold reserves like in those regions, so that other resources for revenue had to be found.
In the Spanish case then a stronger focus was on raw resources to be shipped over; which profited a small group of local elites in Spanish America (esp. Spanish and creole); which meant a system built on highly unjust institutions and accelerating corruption. Whereas the British American case had at least partially transformed product (incl. tobacco or sugar) for export to the metropole; plus by the 17th c. institutions that favored a more democratic development from which not only the elites could profit. I'm also drawing on "Why Nations Fail" by Daron Acemoglu and James A. Robinson here. They sum up some of those points:
It took the Virginia Company twelve years to learn its first lesson that what had worked for the Spanish in Mexico and in Central and South America would not work in the north. The rest of the seventeenth century saw a long series of struggles over the second lesson: that the only option for an economically viable colony was to create institutions that gave the colonists incentives to invest and to work hard.
Social organization
We can also note a more fluid approach at least in colonial Spanish America compared to the later stricter approaches in British America (e.g. with the One Drop Rule). In Spanish America, casta groups included the Spanish at the top of the pyramid, who held the highest posts including that of Viceroy; and "indios", who were under the Crown's protection if they converted, and if noble could continue to hold a certain influence. Others included the "creoles" or Europeans born overseas; "mestizos" of mixed European-native origin; people of African origin; and Asians (often from the Philippines). Then we have a large variety of casta due to mixing between the groups. The important thing to understand here is that this was an *ideal * system which did not actually work completely as envisaged by the Spanish.
Native Americans actually held a special status in the Spanish empire: the legitimization of Spanish expansion officially rested on their conversion of indigenous people to Christianity (they were considered to have no faith, of course). Another big development was the end of native enslavement following Bartolomé de las Casas' influence, starting with the New Laws of 1542. But Africans and people of African descent were not excempt from slavery and usually held the lowest rungs in the colonial hierarchy, as would Asians.
They were usually brought over as slaves, and in contrast to native people there were already centuries of negative stereotypes the Spaniards could build on and continue. In the Americas there was also sexual anxiety of Europeans towards Africans, as seen for example in harsh measures against supposed "slave revolts" - because in many places Africans and "mulattoes" largely outnumbered Spaniards.
Native people (and mestizos and other relates groups) did have possibilites for advancing their position, or even to hide their casta origin and pass for Spaniards in some cases. Recent research highlights how casta categories intersect with esp. class and ethnicity, which is an important point. This can mean e.g. that indigenous people from an elite background could hold many privilieges - incl. having a sword, horse (major rights for Spaniards), and at least into the 17th c. hold onto their traditional lands and rights.
For indigenous commoners however it would have been much more complicated to "switch" groups. Although even people from lower groups (like mestizos or mulattoes) would strategically claim to be from higher groups with success. Similarly, some indigenous groups esp. in central Mexico held more privileges than groups further away from the colonial centres.
So we have at least in earlier colonial times (ca pre 17th c.) still quite of lot of flexibility between these groups. We know of noble Indians and mestizos who described themselves and were perceived as "Spanish". On the other hand, noble Spaniards starting with conquistadors tried to marry into the native nobility in order to gain legitimacy and privileges. R.D. Cope has spoken of "the limits of racial domination" in this context, meaning that even commoners of lower castas had a certain agency and could swith between groups. Nonetheless, he argues that there were long-term effects to the casta system and its exertion of elite Spanish control; and that it would influence people's perceptions of these groups.
We can contrast this with less flexibility and intermarriage between ethnic groups in British America, also in connection with the economic reasons I laid out. This differences I would say even increased with hardening racial attitudes towards native people and African Americans (incl. the One Drop Rule) - would be glad for additions on this comparative aspect. I hope the economic, judicial and social difference between both cases I mentioned do make things a bit clearer. We should also keep in mind that despite such major differences, the British and French also looked closely at Spanish policies in order to avoid what they saw as Spanish mistakes.
Summary: This isn't a question with a clear answer of "A led to B", instead I've highlighted some differences in Spanish and British colonisation in the Americas. a) Pre-contact, modern-day Latin America had a higher population density than the modern-day US. Esp. where major political entities existed like the Aztecs (in central America) and the Incas (in the Andes).
b) Spanish colonisation focused on the right to indigenous labor, christianisation, and the extraction of primary ressources. In contrast, the British the focus was more on acquiring lands, with indigenous people increasingly expelled from them by force. This latter model led to stronger institutions but also stronger segregation in the long run.
And c) major differences in social organization: with the Spanish casta system leading to seperate social groups, but also more flexible (before the 18th c.) than British American systems. In Spanish America intermarriage was even encouraged at least early on with local elites to gain power; and large mixed groups between native, African and European descendants developed over time. Not to paint a too rosy picture though: the casta system continued to influence major discrimation against indigenous and Afro people until today.
In case you’d like to read more about the Mexican case (and excuse the self promo), which I’ve written about on this before:
Edit: Summary
There's also the issue of identification. British communities were designed to be separate from Native communities. For example the encomienda system mentioned by a previous poster doesn't really have a similar style of organization in the British colonies.
Also, the English were very much invested in early processes of otherization as a means of cultural identification. Racial and national identities didn't yet exist in the modern sense, but they still mattered. For example, in contemporary Ireland there were laws prohibiting certain fabrics from being worn by Irish people, as those fabrics were work by English citizens. This carried over in the creation of a distinct sense oof what was English and what was Native, especially in the New England communities. Jill Lepore explores this in detail in her book, "The Name of War."
Also, I recommend Richard White's "The Middle Ground" as an excellent resource on the relationship between the French settlers of their North American territories and the Native Americans living there. The French differ significantly from both the British and Spanish.
Living in the same place is necessary for intermixture, in Spanish America this happened often, not much in English America until much later.
The English Crown never really had a real control in the Americas. During the 16th and 17th century the technology and knowledge to do the trip wasn't really widespread enough and they lacked the experience in colonizing land as well (though they were slowly gaining it with the Irish). The ambition wasn't there neither: most nobles and the crown really didn't realize the potential in investing on the Americas until much later. By when they did catch up they have to deal with the Englishmen which already emigrated and began self-governing long ago there, playing a tacky cooperation/control push relationship with them, and relying on local Englishmen to get anything done until the middle of the 18th century. Even as late as the 19th Century the United States was still gradually acquiring control over its own claimed land! For similar reasons you don't see a big English presence on India or elsewhere until much later.
In contrast, the Spanish Crown was already well capable of doing big overseas projects by the 16th century, and even individual nobles were more than capable of doing private trips without much issue (and so they did, as much of the early colonization was by hand of individual nobles, not the crown itself, though nobles were subservient to the crown directly, which made the power private>public pass more smooth). The experience in colonizing land was already there through the Reconquista of Muslim Holdings in present-day Spain, the colonization of the Canarias Islands and so on, and the Portuguese Sailors in their ships had a lot of experience in colonizing African, Middle East and even Asian ports. The ambition was there as well due to the success of previous colonizations.
This produced a major difference: natives in English Holdings weren't forced (and usually didn't want) to live alongside Europeans, much less being forced to live under their laws and culture, moving a couple kilometers inland was enough to be free (or just live where you always did, most natives lived in months long travel distances from Europeans). In Spanish Lands, natives were forced to live alongside Europeans as no land was free aside the terrible jungle, high altitude Andean locations and Mapuche Holdings (which were in constant conflict and war with the Spanish until the 18th century), which still remains largely native-heritage (little mix) to this date.
Intermixing with people already talking your same language, having your same religion and having a roughly similar lifestyle is a lot easier than intermixing with two totally different cultures.
The Catholic Church and the Spanish Crown (through Church pressure, like with the Inquisitors) invested greatly into converting natives to Catholicism. Meanwhile by the 17th century the Native Americans in the north still largely lived in a similar way they did centuries ago, on Spanish America the natives were living more westernized than traditional lives already and by the 18th century little traces of traditional lifestyle remained. This was done through perverse incentives: for example the Catholic Church always forbid the enslavement of Christians, so this did incentive natives to convert to avoid it. Overall cooperating with the missionaries usually exempted you from the most harsh forced labor and laws.
But there is more. The Spanish also unintentionally did incentive Native American women to want to have kids with White Europeans. Mestizos: kids with white and native heritage, were considered a much better casta in the legal system, for example they were exempted from any forced labor, they indeed had some rights in the legal system (which native kids lacked) and so on. For a native women to have a mestizo kid was a big economic and social advantage, and thus male whites were always flooded by girls. Mestizas also wanted to have kids with the most white-possible folk as that also improved their casta and so on (it reached a point where if a kid was 75% white heritage it was finally considered white criollo in the legal system, the highest a non-born in Europe would achieve, including among whites themselves). Incentives to make the most-white possible kids were all around in the system basically. For a white to have a relationship with a mestizo was relatively easy as both were fully western folks in culture, it was the same than a French dates a French today.
Finally there was a totally skewed sex-ratio (male to female) in Spanish America that didn't exist in English America.
Nobles, soldiers, folks for business, missionaries and so on. If there was a reason to go to the Spanish Americas it was for a work, and not a work in a farm, but a work of any other kind. Since in the past until very recently only men worked, then it was mostly only white men who traveled to Spanish Americas, guys without families, or maybe with a family, but without a motive to travel with them. Aside a couple of asexual folks and maybe a few devoted religious figures, most those guys needed to have sex, and there was almost exclusively native girls and mestizas to do it with, girls which also had direct incentives in wanting to do it as well with white men as explained above! So it wasn't considered culturally wrong in the Spanish Americas to do it with a native girl, or even have kids with her (if you didn't have a family waiting for you, or even if you did most guys would just turn blind eye on each other about it), and as the offsprings were mestizos, as time went this culture just kept growing stronger.
This would continue like this until the 19th century where European families began migrating for land, fleeing from persecution and so on. But then they just assimilated themselves into the culture of the respective nation people (who were already heavily intermixed) rather changing it. Although history books relates this repeatedly, we also have the evidence from DNA that nearly all intermixture happened from white males having sex with female natives and then white-most males having sex with native-most females, until reaching high intermixture levels.
In the English Americas people did travel to emigrate there, religious groups for example, complete families and so on. For Englishmen in the Americas, there never really was a shortage of girls to have sex with (aside in the Old West, but there were far more white girls than native girls by then anyway), and thus they were preferred over natives, which also had a totally different culture, language, and rarely lived in the same place anyway. Even when there weren't those barriers, a different racial system developed in North America, with more emphasis in the pureness of whites and thus a negative perception of mestizos (or of having relationships with natives) as time went on, which still prevails to some extent to this day.
Population density.
Before even Europeans arrived, Central and South America were already far more populated. There was a big death happening after they arrived through infections, war and colonization measures, but when we are talking about the number of survivors by the 17th century, in North America we are talking about a couple dozen of thousands, in Central and South America, a couple millions. However, native (and mestizos with high native heritage) numbers also rebounded much stronger as well because natives and mestizos were integrated with the western agriculture (alongside the fact those people were for the large part also already largely intensive agriculturist), meanwhile still sizeable amounts of the native population in the north kept living in hunter gatherer lifestyle, or non-intensive agriculture, and thus with much smaller population growths until not very long ago. This is reflected in the DNA share: in the United States the native DNA barely makes a 3% of the total share (including native themselves with little mixture). Meanwhile in Latin America the share is over 60%.