Both in terms of a spoken and written language. I'm aware that mastery of classical Chinese was considered a mark of prestige among the educated elite even after Japan stopped emulating customs from Tang -- and that Chinese must've been used to at least some extent for trading in the centuries that followed -- but I'm wondering how common/uncommon it was to encounter someone relatively fluent in the language, oral or otherwise. (In the sense that they could communicate with a Chinese person if necessary.)
I'm particularly curious about how prevalent it was among higher social classes. Would your average courtier/daimyo/imperial family member/etc have been able to speak or write Chinese? For those who could speak the language -- regardless of class -- would it have been more common to know, say, Cantonese or the like (as a result of trading) compared to historical Mandarin/a Mandarin-based dialect?
Classical Chinese was learned as a written language through all these centuries, but hardly ever as a spoken language. This was the case even in the Heian period itself. Bruce Batten writes in his account of a Chinese merchant ship arriving in Hizen (current day Nagasaki prefecture) in 945:
Probably all communications with the visitors were in writing; all Japanese bureaucrats were trained to read and write literary Chinese, but few if any could speak the language. Chinese-speaking interpreters would have been available at Hakata, but probably not in Hizen.
As you assume, there were Chinese interpreters in ports that traded with Chinese merchants. Nagasaki emerged as the natural gateway to Japan for Chinese ships sailing across the East China Sea, and throughout the Edo Period, there was a large Chinese quarter in Nagasaki. Many "Chinese" Nagasaki families lived in Japan for generations, were of mixed Japanese/Chinese blood, and facilitated Japanese-Chinese trade. But spoken Chinese was not a skill that officials or elites in Japan learned or even thought of learning.
During the Edo Period, under the sakoku laws, Japanese subjects were not permitted to leave the country, and Japan had no formal diplomatic relations with China (although Japan permitted Chinese ships to trade at Nagasaki), so there was really no exposure to spoken Chinese, despite Chinese texts continuously entering the country.
This language gap became a problem in the 1860s when Japan decided to tentatively reach out towards China. In 1862, the shogunate decided to send the Senzaimaru, a modern ship they'd bought from the British, on a diplomatic mission to Shanghai. The mission was unannounced, and the participants a rag-tag group of shogunate officials, Japanese merchants, Dutch and Chinese interpreters a Chinese medicine expert, servants, sailors, and a bunch of attendants: determined young samurai from various domains. Some members of this last group were not supposed to be on board. Choshu Domain, for instance, bribed a shogunate official to take their young retainer, Takasugi Shinsaku, along as an attendant. Takasugi was on the trip as an observer for Choshu. He had a keen interest in modern arms and warfare, Western studies, and the effects of Imperialism in China, and kept a very detailed diary of the mission, which researcher Joshua Fogel has used as a main source in studying the Japanese side of the voyage.
The Senzaimaru's arrival in Shanghai in June 1862 was a complete surprise to both Westerners and Chinese in Shanghai, but they were welcomed immediately. These were the first Japanese to set foot in China for hundreds of years, and they were instant celebrities, followed by friendly crowds everywhere they went. Westerners in Shanghai were also extremely welcoming and obliging, especially Shanghai-based missionaries who hoped that Japan would soften to allow Christianity again.
There were a few Chinese interpreters aboard the Senzaimaru, members of those Nagasaki families who'd served that role for centuries. But most of the Japanese mission, particularly the young samurai attendants, made their way around the city relying on their ability to use "brush conversations".
A brush, ink, and a piece of paper could facilitate a conversation, as many Japanese learned on this voyage to their surprise, or a discussion on virtually any subject from the profoundly political to the utterly prosaic. There were minor differences in the way Japanese traditionally used the literary Chinese language, but slight dissimilarities would not have impeded the exchange of ideas. Embedded in the use of the literary language was a wealth of referents rooted in Chinese cultural and literary history that would have been as familiar to well-educated Japanese as to well-educated Chinese
Japanese copies of the brush conversations between Japanese inquirers and their Chinese informants have been preserved, and they cover very complicated, in depth interesting discussions about the Taiping rebellion, English and French colonialism, Russian ambitions etc. To demonstrate the way in which they could communicate deeply by writing, Fogel points out a short exchange between Mine Kiyoshi and Guan Quingmei, both scholars of their respective countries. (p.79)
[Mine] Kiyoshi: You have recently employed British and French troops. Does this not make possible a Shi-Jin disaster in the future? Are the affections of the British and French to be trusted?
[Qingmei] replied: During the crisis last year, we did not bother being concerned about this, as we were planning for the future.
They're discussing the Taiping rebellion, but Mine Kiyoshi is referring to an episode from tenth century Chinese history to explain his point.
it does reveal the easy manner in which educated Japanese and Chinese could communicate. What I have translated as “a Shi-Jin disaster” refers to Shi Jingtang (892–942) of the short-lived Jin dynasty (936–947, in the Five Dynasties era) employing Khitan troops, which led both to disaster for the Jin and to the subsequent founding of the alien Liao dynasty of the Khitans. The message to be conveyed to his Chinese interlocutor was clear: Can you really trust the Western (foreign) barbarians? Their minds and hearts are different from ours. Guan Qingmei effectively ignored the obvious thrust of the question.
So while these educated Japanese couldn't speak Chinese, they could communicate amazingly well without it.
There's one amusing footnote to these brush conversations. They took a lot of time. Time that these scholars had to spare, but not 20th century movie goers. Joshua Fogel finishes his book with a survey of fictional adaptations of the Senzaimaru's voyage. These usually focus on its most famous passenger, the previously mentioned Takasugi Shinsaku, who after his voyage to Shanghai would become a major player in overthrowing the Tokugawa shogunate.
Fogel recounts the making of a WWII propaganda film: a Japanese-created film with a Chinese co-director and Chinese actors, created for a Chinese audience. Noroshi wa Shanhai ni agaru (Signal fires over Shanghai)/Chunjiang yihen (Lingering resentment over Shanghai) with a message of Asian unity against Western Imperial powers. There's one brush conversation in the film, but the directors realized that watching Takasugi and Chinese scholars write long paragraphs to each other would be very boring. Instead, they
introduced a female character by the name of Wang Ying (played by the stunningly beautiful actress Li Lihua, b. 1924, who turns all the Japanese heads when she makes her initial entrance) to act as an interpreter. (p. 181)
In the Kamakura to Sengoku periods, Buddhist communities were the center of Chinese culture and literacy. Many Chinese monks like Rankei Doryu, Mugaku Sogen and Issan Ichinei were invited to Japan, and many Japanese students like Muso Soseki learned their teachings directly. Chinese was widely spoken in Kencho-ji, founded by Rankei Doryu in Kamakura, which later sent trading ships to Yuan. Some of Japanese Budhist students like Dogen and Eisai traveled to study in China.
These monks led a new literary movement of Chinese language, known as Five Mountains Literature. Tōsō-on, Chinese readings brought into Japanese language by them, have strong influence from the Southern dialects.
In the Edo period, spoken Chinese was limited to Nagasaki, with the exception of Manpuku-ji, founded by Ingen Ryuki in Uji, and some Chinese expats like Shu Shunsui and Chin Genpin.
In the later eras, many scholars and poets were eager to study spoken Chinese, particularly among the school of Ogyu Sorai. They learned from Nagasaki translators like Okajima Kanzan, or books written by them.