Everything I have read so far states that Einstein and another fellow wrote a famous letter to FDR stating that the atom bomb was possible and FDR should form a team to research it. After that however, Einstein seems to fade into the background? Why was he not involved more in the project?
There were several reasons.
One to keep in mind in the beginning is that Einstein's initial involvement was not to create a bomb program. The Einstein-Szilard letter just advocated the government form some sort of official way to look into whether nuclear fission might have military implications. This is not the same thing as advocating the building of a bomb, and indeed, the "let's build a bomb" stage of the work did not start until several years later and because of a very different impetus (specifically the MAUD Report). Einstein's role here, at best, was as a famous name attached to a letter — and that letter wasn't quite as important as we like to make it out to be (for obvious narrative purposes). He never advocated building an atomic bomb in any of his letters.
Another is that Einstein's physics is simply not bomb physics. We often overstate the importance of E=mc^2 with reference to nuclear weapons anyway (see here), but even that aside, Einstein's sort of physics was primarily a big picture, abstract, conceptual physics. It was not about the details of nuclear physics, much less the engineering work necessary to actually produce the bomb fuel or design the weapons themselves. He was briefly (and secretly) approached to see if he could be of any help on the gaseous diffusion work in 1941, and it was concluded that he would be of no scientific value.
But perhaps the most important is that on top of all of the above, Einstein was known to be a political radical, "socialist" with a small "s," a pacificist and an anti-racist. Such politics in the 1930s-1950s made him a long-time target of J. Edgar Hoover's FBI, earning him a very large FBI file. He was a political liability.
If Einstein's participation had actually been seen as important to the project, his political issues could have been overlooked. (This is something that often gets missed when people talk about Einstein's exclusion from the work — they did overlook even worse political "infractions" by other project scientists, if the scientists were valuable.) As it was, there was no reason to consider bringing him in on the work, and large liabilities against it.