Nativism and nationalism are related, but the key difference is that the focus of nativist movements is boosting or preserving the status of an ethnic group within an existing state as opposed to other ethnic groups, while the focus of nationalist movements is about changing the nature or position of the state itself (to create new nation-states, to change the political structure of or boost or preserve the status of a nation-state in relation to others). This is with the caveat that these terms cannot be precisely defined, and are often used differently in and out of academia (e.g. 'nationalism' within academia has a more restricted meaning than how it is commonly used outside.)
Nativism is a social / cultural / political movement emphasising the interests of the ethnic group in a country that sees itself as the founding / core group of the society. In the American context of the 19th and early 20th century, this group was the Anglo-protestants, who were losing their demographic dominance of the country by the influx of culturally alien immigrant groups, first the Catholic Irish, then the Southern and Eastern Europeans. Nativism can also be seen in contemporary Western Europe and the United States, and in other recent cases such as South Africa or North East India, for similar immigration-driven reasons. Nativism seeks to bolster or protect the status of this group within the state, for example by restricting immigration of other groups, and by trying to enforcing the cultural dominance of the native group and the assimilation of non-native ones. If a movement does aim at changing the geographical or political boundaries of the state, then it would become nationalist. E.g. a movement aiming to restrict the immigration of non-whites into the United States is nativist, but one aiming at establishing a white-ethnostate is white-nationalist.
There's various definitions of what exactly nationalism is, but it is generally to do with the nature of the state, though it can refer to several different processes. It can refer to changing the boundaries of the state, (e.g. for the Germans and Italians in the 19th century, or the Catalans, Scots or Kurds today), transforming who the state represents (as in the French revolution or Indian independence) or bolstering the status of an existing nation-state against other ones (as in the competing nationalisms of WW1). Nativism and nationalism can go together (e.g. anti-foreigner agitation in times of war), but they can also clash (e.g. when a state's 'official' civic nationalism clashes with more ethnic-based nativism).
The best source I'm aware of on nativism is John Higham's 'Strangers in the Land'. On nationalism there's various classic works, E.g. Benedict Anderson's Imagined Communities, Eric Hobsbawm's Nations and Nationalism Since 1780, or various works by Anthony Smith. There's also a newer work by Azar Gat (Nations) which gives a longer historical perspective.