I can think of four things that distinguished Italians and Irish from other immigrant groups (bearing in mind that Chinese and Jewish immigrants faced different, and in some ways more severe, forms of discrimination):

1. They were predominantly Roman Catholic. The United States had long defined itself as a Protestant nation, and had inherited a tradition of anti-Catholicism that associated Catholics with despotism, ignorance and superstition. Until the mid-19th century, the overwhelming majority of immigrants to the United States were Protestant, but beginning with the Potato Famine Irish immigrants were increasingly Catholic - as were, of course, the Italians. Many of these Catholic immigrants wanted to educate their children separately from the public school system, which in the view of many - including cartoonist Thomas Nast - threatened national unity and promoted a slavish adherence to the Pope.

2. In the case of the Irish, the immigrants tended to flock to the Democratic Party and used their numbers to control the machinery of municipal government in cities like New York and Boston. This political control, in turn, translated into ability to funnel patronage toward members of their own ethnic group - it's not a coincidence that the stereotypical policeman at the beginning of the 20th century was Irish. Many native-born Americans, particularly those loyal the Republican Party and its free-labor ideology, viewed this trend with alarm.

3. In the case of the Italians, a disproportionate number of them were low-skilled and had little intention of staying in the United States. In the late 19th and early 20th centuries, highly-skilled Italian migrants tended to go to South America (especially Argentina), while lower-skilled migrants made their way to the United States (and perhaps as many as a half of them ultimately returned to Italy once they had made enough money). They were consequently stigmatized as a pool of cheap labor that drove down wages, while not making any effort to integrate into their host society.

4. The large waves of Irish and Italian immigrants came during the late 19th and 20th centuries, which was the peak period of immigration in American history. The proportion of the population that was foreign-born was higher than it has ever been in the US. So many Americans, particularly old-stock Protestant Americans, were very concerned that the national character of the the country was under threat from immigrants who did not share the values of their native-born neighbors.