The journalist/diarist Alfred Doten mentions bananas arriving from Hawaii in 1851. This is in California's Bay Area, so we need to put this in perspective if by "in the states" you mean in the East. It is of note that Doten knew what a banana was, so this was not a startling object for him (he traveled from Massachusetts to California two years earlier in 1849), and this is well before the Civil War. From all of this we can infer that people in North America knew about bananas before the Civil War.

Doten's next mention of bananas is in 1872 when he was sent 15 pounds of the fruit while living in Gold Hill/Virginia City in Nevada. From this we can infer several things: since Doten doesn't record every fruit he ate, it appears that he regarded the bananas as remarkable enough to mention. The gap of twenty-one years between bananas does not mean that he had not seen one for that long (I suspect he would have remarked on that as well, given his approach to recording things). Instead, bananas remained fairly remarkable but not unknown on the West Coast from 1851-1872.

I cannot tell you how this translates for the East Coast. Certainly by the time of the transcontinental railroad (1869) the East Coast would have access to Hawaiian bananas, although they would have been an unusual treat. That does not address the early 1860s. Perhaps an authority on the East Coast can address that issue; but it is clear that someone from the East Coast would have been familiar with them.