There's actually a book on this.

Don Van Natta Jr, an investigative reporter at the NYT and later ESPN, wrote a 2004 book First off the tee on Golf and the American presidency.

This book includes the observation that since Taft we've had 3 non golfing presidents (Hoover, Truman and Cater).

[side note: FDR's polio obviously was a huge hindrance to his being able to golf but he was an avid player before stricken with polio. Thus FDR is included as a golfing president/a president who golfed for fun/relaxation/pleasure]