I can answer this question in reference to one of the most important Southern Yankees, George Thomas. Thomas was a long-time Virginian officer of the US army who, upon hearing of Fort Sumter and the subsequent secession of Virginia, was forced like many of his comrades to break for either the North or the South (the vast majority of fellow southern officers broke for the South, with especially notable exceptions including Winfield Scott and David Farragut). Thomas was the golden scion of his southern Virginia plantation family, being a decorated veteran of the Mexican-American War and seen as an exemplar of a Southern gentleman, much like Lee.

However, when he decided that his oath to defend the government came before his loyalty to Virginia (he didn't really think highly of the concept of states' rights) and volunteered his services to Lincoln, his family's attitude suddenly shifted dramatically. His two sisters, who minded his plantation, turned his portrait to the wall and essentially denied ever having had a brother named George, and this attitude persisted past Thomas' death in 1870.

Confederate reactions were similarly hostile, with Jeb Stuart stating that he would like to hang the, uh, traitor from a tree (self-awareness was not a strong suit of the Confederacy). Stonewall Jackson denounced him. These men would not survive the war, but the sentiment persisted among the Southern writers who would form the corpus of The Lost Cause, which really really did not like Thomas' support for freedmen in his postwar military district in the northwestern part of the former Confederacy (incidentally, the KKK would have its first incarnation in Pulaski, Tennessee during Thomas' military rule, and Thomas tried to stamp it out from the beginning). The Lost Cause sought to portray the southern cause as inherently noble, so Thomas was seen as basically turning his back on his home state to support an oppressive war of aggression by the North. The Lost Cause has a notorious eastern theater bias, so not as much ink was spilled as could have been, but suffice to say, they viewed Thomas as even more base and evil than the Yankees who fought with their stated.