Since the vast majority of the war happened in the South, it was usually only Southerners that were "refugees." It's really hard to give any definite answers to your question, since there were considerable variables for every situation. Sometimes a family would move from town to town to try and escape the advancing Federals, or sometimes they would pick up and leave from one state to the next (I'm thinking of Sarah Morgan and her family - they left Louisiana and resettled in Texas). Some Southerners left the country altogether, either during or after the war, and resettled in Mexico, Brazil, France, the Caribbean, and even Britain. There is actually a very large festival held in Brazil every year for the descendants of the 20,000 Confederates who immigrated called the Festa Confederada.

There was quite a bit of "refugee" movement in and out of Missouri and Arkansas during the Civil War, mostly due to the extreme guerrilla activity in the region. Many people moved a long their political alignments, whether pro-union or pro-secession. At the beginning of the war, soon after Arkansas had seceded from the Union, some pro-union whites fled the state into the southern area of Missouri, particularly around Springfield, which was a pro-union town. In northern Missouri, in the area known as "Little Dixie" (along the Missouri River), pro-union whites left that area in droves as well, going to Iowa, St. Louis and other parts of Missouri to try and get away from the pro-Confederate guerrillas, called "Bushwhackers", who were extremely violent toward pro-union men and their families. On the other hand, you had pro-secessionist whites moving from these same areas into central and southern Arkansas and across Texas. They did this for two reasons. First, for slave owners to be able to protect and keep from losing their property and their lives to the Federal army and from pro-union guerrillas, called "Jayhawkers". Jayhawkers were extremely violent toward pro-slavery men and their families. Secondly, they moved away to get out of the way of military violence (battles) and military rule (martial law, etc.) of what they saw as an occupying force. Lastly, probably the biggest single migration of people from this area occurred August 25, 1863. General Order No.11 was issued to 4 Missouri counties that bordered Kansas. In summary, this ordered the civilians living in these 4 border counties, to pack up and leave the county within 15 days. This was in response to the August 21 raid on Lawrence, Kansas, a Jayhawker stronghold, by pro-Confederate guerrillas, which resulted in the town being burned and the killing of 164 civilian males. As the war continued and things started to improve, you did see some people return to their homes (if they were still there), but many, if not most, did not.

I can't speak directly to the size or movement of refugee groups, but white Southerners moved about in large enough numbers that the CSA made arrangements to facilitate their voting when they were displaced. I found out about those in Kenneth Martis' Atlas of the Confederate Congresses and wrote about it a bit previously.

Many enslaved people also chose to displace themselves when given the chance, of course, but they're not really refugees from the conflict in the same sense. Rather they're refugees from the centuries-old conflict that ultimately brought the Civil War about. I'm afraid I know less than I'd like about their wartime circumstances, save that many of them ended up as contraband labor for the United States. For some that turned into conventional soldiering, if under white officers and at less pay, once the law permitted their enlistment.