Latin was the practical choice for the negotiations and consequently for the treaty text itself. It was also what could be called the universal, standard choice of the time.
Having no knowledge of each other's languages, the two negotiating parties had to use translators. The Russians had a young Polish catholic priest Andrei Bielobocki as their translator, and the Qing side's translators were Jesuits Jean-Francois Gerbillon who was French and Thomas Pereira who was Portuguese. Although they came from three different countries, Latin was a language the translators knew well, since they learned it as the fundamental part of clerical curriculum.
This fits in the wider picture of the world diplomacy in the 1600s. Latin was the lingua franca of diplomacy of that era. Even though it had been already an extinct language in daily life, Latin was well alive among clergy. And clergy supplied most of the diplomats at that time. Anywhere where European conquerors and merchants went, clergymen followed. While their primary objective was to establish missions and convert inhabitants of the distant lands, they often found themselves employed by foreign leaders as advisors and envoys. Considering this, it is no surprise that the letters sent by the Kangxi Emperor to the Russian Tsar that initiated the events leading to the Treaty had also been written in Latin.
Let's go back to the language versions of Treaty of Nerchinsk for one more interesting linguistic note. While the Latin version of the Treaty served as the authoritative one, translations into Manchu and Russian were also produced. Manchu is the curious one here. Trying to reach agreement with the Russians, the Kangxi Emperor allowed for breaking of a long-standing rule of Chinese diplomacy that the other party is always regarded as the inferior, submissive one in any treaty. Going against this rule risked opposition from the conservative Qing Empire apparatus that was predominantly Han, with Chinese as its official language. Therefore, through the use of Manchu instead of Chinese, the expected fallout within the Empire's bureaucracy was mitigated.
Sources and further reading:
Eastern Destiny: Russia in Asia and the North Pacific by G. Patrick March (1996)
Boundaries and Trade in the Early Modern World: Negotiations at Nerchinsk and Beijing by Peter C. Perdue (published in Eighteenth-Century Studies, 2010)
The circumstances of the negotiation of the Treaty of Nerchinsk are fascinating, and the fact that the primary language was indeed Latin is worth exploring. Now, there is an answer here by /u/SalTez, but I have issues with some of their framing, so take this as a companion piece.
If we accept that Latin did in fact serve as an intermediary language due to fluency on both sides, then the question of 'why Latin?' entails an implicit question in the negative: why not Mongolian? Both the Qing and Muscovy-Russia had expanded into Inner Asia in part off the back of alliances with Mongolic and Turkic petty rulers, and by 1689 both sides would have had plenty of fluent Mongolian speakers and Mongolian-writing clerical staff. When you bear that in mind, Latin again seems like the less intuitive option. The reality is not that Latin was convenient, but that Latin was useful specifically to the Jesuits, who had secured the trust of the Qing court. The Jesuits firmly asserted that the Russians' Mongolian translators were incompetent, and consistently protested against any attempt by the Russians to circumvent them by sending communications in Mongolian. While this could not prevent these apparently illicit messages from being sent, the court allowed itself to be swayed by the Jesuits and denounced the attempt to create Mongolian-language back-channels during the negotiations. Keeping all communications in Latin allowed the Jesuits to monopolise the process, something the Qing court was happy to indulge.
This is not so much a causal factor as a symbolic point, but it is worth also considering what the function of the Treaty of Nerchinsk was. You see, the Treaty of Nerchinsk was not simply the settling of a border agreement in outer Manchuria, though it certainly was part of it. In an implicit sense, it entailed the partitioning of Inner Asia into spheres of influence: one Russian, and one Manchu. Yet the people within the Qing part of said partition, the Mongolians, are absent from the text of the treaty, both in the sense of its substance and in terms of the language it was written in. Had the negotiators on the Qing side been Mongolian, how likely would they have been to sign away their own sovereignty by implicitly accepting that it was the Qing who determined if they were under Manchu suzerainty? The Jesuits, on the other hand, would have been direct representatives of the imperial centre, with no particular attachments to the peoples of its Inner Asian frontier. While it was the Jesuits themselves who insisted on Latin, rather than the Qing court, in doing so they still helped to reinforce the status of the Mongolians and Mongolia as what Perdue calls the 'excluded middle' in the treaty.
What I take some umbrage with is the notion that a translation into Manchu was somehow unusual. Firstly, Manchu was still the empire's prestige language and retained that status – if in an increasingly shared role with literary Chinese – down to the overthrow of its ruling house in 1912. By 1689 the Qing were barely half a century post-conquest, and they had also just put down a major Han Chinese revolt (the Three Feudatories uprising) at the start of the decade; this was not the sort of environment in which an air of conciliation towards Han Chinese conservatives was really in the cards. Moreover, at this point, frontier affairs in Inner Asia were the exclusive purview of the Banners, which did include the so-called 'Martial Han', but which were in the main part the organising force for the empire's Manchu elites. The Kangxi Emperor could not give less of a toss about the Han Chinese bureaucracy, because affairs in Inner Asia were not under their jurisdiction: frontier matters were Manchu matters.
Peter C. Perdue, China Marches West: The Qing Conquest of Inner Eurasia (2005)
Peter C. Perdue, 'Boundaries and Trade in the Early Modern World: Negotiations at Nerchinsk and Beijing', Eighteenth-Century Studies 43:3 (2010)