One of the most defining causes for European immigration during the American Civil War and postwar period was the promise of land.

In the prewar era the United States expanded with the Louisiana Purchase, Texas annexation, Mexican cession, and Oregon Treaty more than doubling the size of the country from 1800 to 1850. The United States now had vast amounts of land and with the 1830 Indian Removal Act mandating Natives move west of the Mississippi and 1851 Indian Approriations Act forcing Natives onto reservations it was prime real estate for white settlers.

Prior to the Civil War there was a strong desire to settle this land among the northern states but there was opposition from wealthy plantation owners in the south who feared their land and agriculture production would be devalued. After the outbreak of the war and withdrawal of the southern states from the Union congress passed the Homestead Act of 1862 which authorized free (yes, free!) land to anyone willing to settle on and farm the land for a period of time. Antebellum Agricultural Reform, Republican Ideology, and Sectional Tension by Sarah T. Phillips is a great read on the politics of land in the mid century period.

The chance of land ownership was incredibly enticing to the poor of Europe, especially in Ireland and Germany which were experiencing significant hardship during this time.

In the 1800's Ireland was under the rule of the United Kingdom and was brutally oppressed both religiously and economically. Essentially most of the land was owned by Protestant English landlords and the crops, livestock, and wealth was being sent to the UK. The Great Potato Famine of the 1840's was a culmination of this. With the allure of land ownership and high paying jobs the Irish Catholics saw the opportunity in America and came by the millions.

Over in Germany there was a similar story. After the conquering of the Holy Roman Empire by Napoleon and his subsequent defeat the German States were in shambles. In 1815 the various states created the German Confederation as a loose association between them but it failed to address the issues of the people and by the 1850's there were revolutions attempting to overthrow the wealthy ruling elite. Once again the promise of land and freedom in America was a huge draw to the German immigrants.

Land ownership was not the only thing attracting these immigrants. There was also the chance to earn high wages. Due to the destruction from the war and the westward push of new settlers there was a growing demand for consumer products. The industrial revolution was well under way and factories had a shortage of workers. Therefore factory wages were through the roof compared to their European counterparts, though this gap would subside by the late 1880's.

This should not be taken as a definitive answer, and by and large I'm not specifically going to answer the part of the question about what drove European immigration to the US in the 1860s - I will leave that to a more knowledgeable commenter.

But specificially around some of the assumptions in the question, I can speak a bit to these.

Firstly, the fighting and violence of the Civil War was almost completely restricted to the American South, and even there it was heavily concentrated in a few regions only, most notably Virginia, Tennessee and Georgia. Much of the conflict even in other areas of the Confederacy was relatively short (for instance, the largest city in the Confederacy, New Orleans, was captured after essentially a brisk two week campaign), and whole swathes of the Confederacy for all practical purposes never even saw armies marching across it.

Outside of the South, the war really had little if any direct impact on daily life, being more like a foreign conflict that a domestic one. California and the Pacific Coast were far, far removed from the fighting, and only with the brand-new in 1861 transcontinental telegraph could even receive same day news about the conflict.

With that said, there were significant numbers of refugees from the war - or, more correctly they would be what we would call in modern parlance "Internally Displaced Persons", as refugees are technically international. Namely, this would include the so-called "Contrabands", ie slaves who individually or in groups escaped and found their way to Union lines and camps. Overall perhaps something like 500,000 out of four million slaves became Contrabands.

A few further points around the war and immigrants - overall, the Union was not terribly interested in directly recruiting immigrants, especially Catholic ones from cities, in part specifically because authorities didn't want the bad press this could cause among potential future immigrants in Europe. Immigrants did serve in the Union (and Confederate) armies, of course, but they were disproportionately underrepresented. On top of this, despite the notoriety of the draft, it was a fractionally small percentage of the Union army (something like 2% of the total being conscripts), and overwhelmingly a volunteer force. Therefore, from the perspective of an immigrant to the US, and here it's worth noting that immigrants overwhelmingly went to Northeastern cities or to the Midwest, the Civil War was mostly in a distant region of the country, and fighting in it was a mostly voluntary affair. More specifically on Irish immigrants and Civil War service here.

With that said, immigration to the US did decrease during the Civil War, hitting a nadir of about 95,000 arrivals in 1862 (the lowest in about 20 years). It's worth noting, however, that this was a continuation of a trend starting in 1857, when a severe economic downturn in the US made immigration a less attractive option for many. The arrivals figures quickly rose with the end of the conflict, however.