Why did the Union leadership, after finally winning a long bloody war, have any desire for reconciliation?
Why, in the terms of the time, was it not seen as obvious and simple that the entire Confederate leadership should be tried for treason?
From the perspective of 150 years later, it seems obvious that the entire Southern officer corps could and should have been lined up against a wall and shot, but clearly I'm seeing the past through the lens of present-day ideas and politics.
Essentially, the answer to each of the questions you asked comes down to one key concept: political realism in the context of a fragile peace. Essentially, this was accomplished by means of a "no hard feelings" campaign.
Why did the Union leadership, after finally winning a long bloody war, have any desire for reconciliation?
Punitive measures against the Southern leadership class and against the South more broadly would have endangered an already delicate peace. Congress had already eliminated slavery via the 13th Amendment, and in so doing eliminated the cornerstone of Confederate society and Southern culture. Trying the South's political leaders for treason (even if the rank-and-file were absolved of charges despite participating in treason as well) might well have reignited hostilities and prolonged the bloodshed. The war had already claimed hundreds of thousands of casualties -- I remember once figuring that based on one estimate, one out of every ten males of military age at the time was a casualty -- and neither Washington's political class nor Northern public opinion would have had any interest in doing that. While there were political forces who took a punitive mindset, many others wanted to get on with what was certain to be the difficult task of reintegrating the South back into American society.
Why, in the terms of the time, was it not seen as obvious and simple that the entire Confederate leadership should be tried for treason?
In the immediate aftermath of the War (and indeed, for decades later) such accusations would have reignited tensions and made the North's military occupation of the South (already far from incident-free) all the more difficult. On top of that, Art. 3 of the 14th Amendment stripped participants in the Rebellion of the vote (temporarily) thus ostensibly neutralizing the political threat they might pose.
Lee in particular would have been politically explosive to touch. Whereas Jefferson Davis was very unpopular by the end of the War, Lee maintained a tremendous amount of loyalty among the soldiers he commanded and was a much more popular figure throughout the South. For that reason, it was important for Lee to represent the face of national reconciliation post-bellum, and that end would not have been served by trying him for treason.
As lee himself wrote shortly after the conclusion of the War
It should be the object of all to avoid controversy, to allay passion, give full scope to reason and to every kindly feeling. By doing this and encouraging our citizens to engage in the duties of life with all their heart and mind, with a determination not to be turned aside by thoughts of the past and fears of the future, our country will not only be restored in material prosperity, but will be advanced in science, in virtue and in religion.
150 years on, it seems easy to say that justice would have best been served by laying down the hammer on the South, but that's the bias of the present speaking. In a society which bore the scars of the war, many political leaders would have a very different outlook. Signing letters of the mothers of the fallen is not a pleasant business.