E. B. White had serious anxiety and pretty much shunned public appearances after getting flustered and embarrassing himself at a PHD thing at Dartmouth in 1948:

Public gatherings—and most private ones, as well—made him jumpy. For years he had passed up family weddings and graduations, town meetings, dedications and book awards, cocktail bashes and boat gams and garden parties. As his literary reputation widened when he was in his forties and fifties, he did make it to a few select universities to receive honorary degrees, but despite prearranged infusions of sherry or Scotch he found the ceremonials excruciating. “So the old emptiness and dizziness and vapors seized hold of me,” he writes to my mother after his honoris causa Ph.D. appearance at Dartmouth in 1948. “Nobody who has never had my peculiar kind of disability can understand the sheer hell of such moments, but there they are.” And when the time came for the encomiums and the enrobing, there in the sunshine at Hanover, he went on, his hood—“white, quite big, and shaped like a loose-fitting horse collar”—became entangled with the honoree in the next seat, Ben Ames Williams: Andy’s worst dreams come true. “When I got seated the thing was up over my face, as in falconry,” he continues. “A fully masked Doctor of Letters, a headless poet.” After that, he stayed home, even passing up an invitation in 1963 to go to Washington and receive the Presidential Medal of Freedom from Lyndon Johnson.

-"Andy" by Roger Engell for The New Yorker February 14, 2005

So, a reason for decline, but one he'd most likely have wanted to keep secret.

​