We are familiar with the idea that both were heavily criticized by much of the mainstream at the time for being pointless, immoral, or iconoclastic. But did these two movements see each other as working together to take music in new directions? Or did they antagonize each other?
I can not speak of greater influences between the two but can talk a bit about a later musical exchange for the 60s: the influence of avant-garde music - including Terry Riley - on the later Velvet Underground musician John Cale. I should note that the influence of drones became an integral part of the Velvet Underground; but probably the strongest influence came from Andy Warhol's Factory scene, as argued here in a fascinating post by /u/hillsonghoods (I'm adapting this from my answer in that thread).
At the same time, it does not seem that the exchange between rock and avant garde classical music has been the norm -- for a long time these scenes did not intermingle regularly. Even still into the 80s, musician Arthur Russell has talked about the difficulties of building connections between the classical and more popular "musical worlds" in downtown NY.
Cale had studied classical music in England and was particularly interested in avant-garde composers like Webern. At the same time, he exchanged views with the English composer Cornelius Cardew, and met the electronic music pioneer Xenakis in the USA, where he traveled on a stipend. So he had a strong background in modern composition before coming to New York. There, he continued to be interested in classical/avant garde music: he early on participated in a piano concert of a work by Erik Satie, directed by John Cage, that lasted several days.
One main reason for John Cale to come to New York was to meet avant garde composer LaMonte Young, and they soon started collaborating. In ca. 1963 the group called alternatively the Theatre of Eternal Music or the Dream Syndicate (a bit confusingly: not to be mixed up with the 80s rock band!) was started with Young, Cale, Marian Zazeela, Angus MacLise, and Tony Conrad. Conrad, Cale and MacLise lived in the same building for a while. However this was not a fixed "band" line-up but rather a changing roster of participants added to this core that also included later "Fourth World" music pioneer and Brian Eno collaborator Jon Hassell and Terry Riley among others. Of these members we have both Cale and MacLise leaving to join the Velvet Underground in late '65.
One major influence of especially Cale's Dream Syndicate (below called DS) experience on the Velvets would be his playing of drones on his viola. This together with Lou Reed sometimes using an open guitar tuning, aka "ostrich" tuning as per the Ostrich Song, helped to form their avant-garde and somewhat mystical rock sound.
In her book Fluxus Experience Hannah Higgins describes Young's influence on Cale and his drones in more detail:
Collaboration also marked musical experimentation. LaMonte Young, for example, helped Cale turn his viola into an experimental, and then a rock, instrument. Called the Dream Syndicate, their project of 1963-64 relied on two electronically amplified voiced and an amplified violin and viola. The idea was to hold the same notes unbroken for two hours; for example, Young might hold the deepest bass note, Cale the middle three tones on the viola, and Marian Zazeela (Young's wife) the final, high-pitched note on the violin. Of this experience Cale commented: "That was my first group experience -- and what a one! It was totally new! I mean, tapes as art objects! ... The indians knew this monotone... But they use totally different sounds.
Shortly thereafter, Cale brought static music into pop culture (with a later member of the Dream Syndicate, Angus MacLise). The droning viola undertone of their band, the Velvet Underground, derives from the Dream Syndicate.
Cale mentions here the adaptation of drones as used in traditional Indian music (and other regions), e.g. through tampura, to the minimalist avant-garde music of the DS, creating something different in the process. The other element of using long durations is also something used in many cultures, incl. African traditional drum circles, often to induce trance states.
LaMonte Young and other musician like Terry Riley, who also sometimes played with the group and whom you mention, would build on this idea to host all night droney concerts. But at this earlier stage we can see how the Velvets managed to adapt such drones and long-ish durations to their more pop oriented work, and how they influenced their use of discordance and feedback.
DS tracks from 1963 like B flat dorian blues could be dominated by Young's soprano saxophone; this longer performance from '65 probably is a better example of their drones and feedback use (note: it gets a bit calmer after about 10 mins., but none of this is really easy listening). While of course very different in approach, there are parallels esp. with the feedback-y guitar/viola in the latter part of Velvet Underground's Heroin; and in their little known first EP Loop created by Cale.
Cale and Young give some more insight into their work process here (taken from Please Kill Me: The Uncensored Oral History of Punk, by Legs McNeil & Gillian McCain):
John Cale: [...] I first met Lou at a party and he played his songs with an acoustic guitar, so I really didn't pay any attention because I couldn't give a sh#t about folk music. I hated Joan Baez and Dylan -- every song was a f#ing question! But Lou kept shoving these lyrics in front of me. I read them, and they weren't what Joan Baez and all those other people were singing.
At the same time, I was playing with La Monte Young in the Dream Syndicate and the concept of the group was to sustain notes for two hours at a time.
La Monte Young: [...] John Cale started playing with my group, the Dream Syndicate, which literally rehearsed seven days a week, six hours a day. John played specific drone pitches on the viola -- until the end of 1965, when he started rehearsing with the Velvet Underground.
Another interesting part of this is Cale describing being first drawn to Lou Reed more by the writing than by the (still acoustic) music. It's probably too easy to just divide the VU's influence into "Cale=experimental" and "Reed=pop, great writing", as both of their later solo careers have shown. Not to mention that it would discount Tucker's and Sterling's great contributions. Nonetheless I think it's helpful to keep musical lineages in mind running from the DS to VU, in addition to the Warhol/Factory influence in the band's visuals, ligthing, lyrics etc.
While the latter was probably more contemporaneus, Cale did continue to work with LaMonte Young on his long form compositions for some time. And both John Cale and Lou Reed would go on to use drones and feedback to great effect, for example in Cale's production of the Stooges and in Reed's infamous Metal Machine Music of '75. Even after the time with VU John Cale would once work with Terry Riley, for the album "Church of Anthrax". According to Cale, Riley disapproved of the result as he was relatively low in the mix. It remains an interesting mix of avant rock and electronics nonetheless, see here for the title track.
One problem with attempts to look deeper into DS/Young influence is that for a long time their recorded output was withheld by Young himself. According to Tony Conrad because it did not hold up musically to the legend that had by this time built around the DS and Young in particular. Luckily in the last years more recordings including some with John Cale and Angus MacLise have resurfaced. Despite all this the group continues to be namechecked by artists doing experimental, noise and minimalist music -- similar to VU's enduring influence in the rock realm.
For John Cage (not Cale) I only came across a small discussion by David Toop (in Into the Maelstrom: Music, Improvisation and the Dream of Freedom, Before 1970). Toop argues that Cage was much more interested in Japanese art and literature than in the Jazz and Free Jazz played around the corner from him in NY. I would suppose this could apply to Cage's views of rock'n'roll or later Rock too, but hopefully others can add to it.