Short answer: yes, there were concerns. But the phrasing "terrorism, political violence, or sexual violence" is showing a bias for contemporary concerns and political rhetoric. I'm gonna give a shortish answer. This question is easily answered in any standard textbook or google search.

Whenever there is a large influx of immigrants into any place, there are concerns of how it will negatively impact that place. OTOH, there's also people willing to offer assistance and support and help people adjust.

The most pressing concerns were the Catholicism of Irish, Italians, and others. The United States began as a deeply anti-Catholic nation. Ironically, that's the largest religious denomination today. Catholics were viewed as inherently disloyal to the USA and blindly loyal to the Pope. They also tended to see Irish and southern Europeans as inherently lazy and more preoccupied with leisure than work, unlike protestants of Britain and northern Europe - the "good whites".

There were also concerns of radical politics, particularly socialism or anarchy. This was prior to the successful Communist Revolution in Russia, and Marx had predicted that it would be highly industrialized nations such as the US or Germany that would have a revolution. Many of these immigrants swelled the ranks of the working class. They did hard labor for low wages with no safety regulations. While many sought to improve their condition within the law, there was considerable support for more radical forms of protests. Many of the strikes would be more accurately described as a war or a battle. And those that tended to work within the law were skilled workers that had leverage, unlike unskilled workers that were easily replaceable by even cheaper labor, such as African-American or <insert different European ethnic group> scabs.

There were also many that helped immigrants and the working class in general. Such as two of my heroes Jane Addams and Mother Jones.