Newton Knight was a complex pro-Union Southerner. Knight lived in Jones County in Mississippi where a number of folks opposed secession. In fact, they even elected an ant-secessionist to represent them at Mississippi's secession convention. (This candidate ended up succumbing to political pressure, switching his vote.) Knight originally fought for the Confederacy, was discharged a year later, rejoined, and then went AWOL. Knight had grown increasingly despondent with the Confederacy--for example, they seized his home and he was outraged when the Confederacy allowed wealthy landowners--one white male per 20 slaves--to skip military service. The CSA army arrested the AWOL Newton, and they destroyed his farmstead and some say they tortured him. He eventually escaped. As the war continued to turn dire and as Confederates began deserting more and more, Jones County became a haven for deserters. In late 1863,the deserted formed the Knight Company, electing Knight as their captain. Knight led his company in fourteen skirmishes against the Confederate army. Local legend says they actually seceded from the South, forming the Free State of Jones. After the war, Newton divorced his wife in the 1870s. He then, despite the miscegenation laws that were on the books since 1822, married Rachel, a former slave Newton's grandfather owned.

See Victoria E. Bynum's "'White Negroes' in Segregated Mississippi: Miscegenation, Racial Identity, and the Law" in The Journal of Southern History 64.2 (May, 1998)

Yes, there were Union "guerrillas" operating in the [Trans-Mississippi] (http://en.wikipedia.org/wiki/Trans-Mississippi_Theater_of_the_American_Civil_War) Theater of the war, largely along the Kansas/Missouri border areas.

In fact, the term "Jayhawker" which is known better known as "Jayhawks" as in the Kansas State Jayhawks were Union guerrillas who performed raids and attacks in pro-slavery territories of Kansas and Missouri.

However, in reality, most of these groups, both Union and Confederate used the excuse of the Civil War to basically become armed bands of thugs and thieves who operated under the guise of being partisans, when all they really wanted was money and loot.

Not sure if it counts as guerilla warfare, but the Loudoun Rangers was a union cavalry unit composed of disaffected Virginians, tasked with raiding their home county. Some of them came from Cole's Maryland Cavalry, which was a similar unit of border raiders. One of their main foes were their relatives in a similar independent raiding unit, the 35th VA cavalry.

Contrary to the source I linked, which says "Like the Loudoun Rangers, the 35th Virginia had been raised for the specific purpose of "ranging in the border counties," and the men never resigned themselves to being forced to follow the main army into distant regions in violation of their special enlistment contract.", the Confederate unit was the first into Gettysburg before the battle, and made a little girl cry by stealing her pet pony. (page 85 of Trudeau's Gettysburg)

In Texas there were numerous German immigrants, some of whom were associated with the liberal and even socialist revolutionaries of 1848 before migration. They were generally pro-Union, and raised their own militias to protect themselves (presumably from the Confederacy) during the Civil War. However, they did not actively engage in attacks against he Confederacy.

There is at least one incident where German militiamen from the Texas Hill Country attempted to march to Mexico, and make it to Union territory from there. They were ambushed by Confederate soldiers along the way and never made it.

I'll see if I can dig up a source on that. I remember a really good one available online but don't have it bookmarked or anything.