Yes, but I can only find mention of one visit.

After her lobotomy in 1941, Joseph P. Kennedy put his daughter in a psychiatric hospital in New York. In 1949, he had her transferred to a Wisconsin home for developmentally disabled people, run by nuns, called Saint Coletta. Publicly, Joseph and Rose (her mother) essentially erased Rosemary from the Kennedy narrative, and privately discouraged questions about her whereabouts from their other children. However, the siblings did somehow get information about Rosemary eventually, as Jack (John F. Kennedy) visited his sister at Saint Coletta in 1958. According to Kate Clifford Larson's biography of Rosemary, his sister's condition shocked Jack and caused him to take a newfound interest in disabilities legislation. When he became president, JFK established the National Institute of Child Health and Human Development and the President's Panel on Mental Retardation. In 1963, he gave a special address to Congress on the issue, and on October 24, 1963, he signed the Maternal and Child Health and Mental Retardation Planning Amendment to the Social Security Act. (A month later, of course, he was assassinated.)

JFK was greatly encouraged in those efforts by his sister Eunice Kennedy Shriver, who had been the sibling with the closest relationship to Rosemary. After Joseph P. Kennedy's debilitating stroke in 1961, Eunice took over Rosemary's care and in 1962, went public with Rosemary's story. She wrote an article for the Saturday Evening Post, "Hope For Retarded Children" and in 1968 founded the Special Olympics.