In a word, Longstreet was a realist. He accepted that the South has lost, and was looking for the best way for her to bounce back. But while he wasn't alone in the former officer corps in his caution "to accept the terms that are now offered by the conquerors" following the implementation of Reconstruction in 1867, he was fairly unique in his advocacy for actual cooperation, which resulted in considerable vilification by Southern veterans and writers over the next few decades. Maybe if he had kept is views to himself it wouldn't have been so bad, but he put them in a letter that was published in a New Orleans newspaper. I don't know if the entire text is available, but "Lee's Tarnish Lieutenant" has a fairly extensive quotation. In short though, He was arguing that cooperation with Republicans was essential in order to best be able to mitigate the adverse effects of Reconstruction - "if whites won't do it, the thing will be done by the blacks" - and additionally that ensuring a Southern presence within the Republican party was essential to limiting the ability of the newly enfranchised African-Americans to have any real power with their vote.

As you can see, his views are still fairly offensive as far as our ideas of racial equality goes, but for a Southern audience, still smarting from defeat, and still pretty damn racist, he might as well have just waved a white flag. Most of them weren't willing to give even an inch, and of course, as the next few decades would bear out, the South was fairly effective in ensuring the failure of Reconstruction, and the continued subjugation of the African-American population under Jim Crow. It didn't matter to them that Longstreet firmly believed he was advocating in Southern interests, and for the continued marginalization of the Black population at that. The Republican Party was the enemy. It was everything that stood in opposition to white, Southern civilization. One of the most core aspects of the Southern views on their defeat was to ensure that their honor remained intact - defeated on the battlefield but not in spirit. Longstreet's path went against that, however much his long term view of the continuance of a white dominated South may still have aligned.

Longstreet perhaps could have defended himself, but he simply never really tried, at least in the early days. A few private letters exist which speak to his commitment to white supremacy, but he never made strong, public statements to that effect in order to clarify his position. It didn't help that within a few months, he was granted his Federal pardon, which would allow him to again run for office, and of course led to accusations of abandoning the Confederate cause out of sheer self-interest.

He then just keep digging that hole deeper, endorsing Grant for the presidency, and then accepting a Federal job in the Port of New Orleans. This just only continued to feed Southern attacks on his generalship and character, and soon enough, Longstreet was essentially the sole cause of Southern defeat, having been made the lynchpin of defeat at Gettysburg, and in turn Gettysburg the lynchpin of defeat in the war itself. It was essentially a vicious cycle, with each side acting and reacting to further entrench the other's position. When, in 1896, he published his memoir and dared speak an ill-word of General Lee in defense of himself, well, he might as well have taken a dump on Jesus Christ himself as far as Southern audiences were concerned. While he wasn't exactly at Sherman's level, Longstreet had very much come to be a villain of the 'Lost Cause' narrative as it was formed in the late 19th century.

James Longstreet and the Lost Cause by Jeffry D. Wert, in The Myth of the Lost Cause and Civil War History, edited by Gary W. Gallagher & Alan T. Nolan

Lee's Tarnished Lieutenant by William G. Piston