In the Shadow of the Moon (the book, not the movie) touched on the issue of infidelity a little bit. The book didn't really name names or go into anyone's specific details, but it said that 1960s astronauts had a lot of opportunities to cheat on their wives and many of them did so. Frank Borman, Al Bean and Mike Collins were noted as being a bit unusual because they were totally faithful to their spouses.
There is some disagreement about how much this kind of behavior was tolerated by NASA at the time. There is a widespread belief that Donn Eisele was never allowed to fly again after Apollo 7 because he had an indiscreet affair and then divorced his wife. In his autobiography Deke Slayton says this is nonsense and that he had no intention of playing morality police, and it was Eisele's performance/lack of effort in training that was his undoing. It's impossible to know how true this is since we're relying entirely on Slayton's claims and he was never particularly open about how he picked crews.
Source?