I'm not sure who the first people were to enter China and attempt to learn the language. As far as I am aware, the first real attempt made by a foreigner to learn Chinese was by Michele Ruggieri in 1579 and Matteo Ricci in 1582.
I know that missionaries were able to show up in Macao before that, but I'm not sure how much Macao used the Chinese language at the time.
Anyways, despite all this, how would the first people to explore China have learned the language? How would they have been taught the language when there are some incredibly difficult ideas to communicate with the writing system the Chinese had.
Things for example like 日 I imagine would have been easy, then 月 as well, 山、川、口、(sun, moon, mountain, river, mouth respectively )would also have been easy. But what about stuff like 混亂 (confusion), 愛 (love), etc. How would these things have been communicated to the foreigners?
This has been asked a number of times, and you can see some of the past discussions at the following posts:
As an aside, there are some misconceptions your question is based on. Specifically about these points:
Anyways, despite all this, how would the first people to explore China have learned the language? How would they have been taught the language when there are some incredibly difficult ideas to communicate with the writing system the Chinese had.
The written language did not correspond to the spoken language in the time that the travellers you mentioned would have been there. The speech of someone in Beijing (Mandarin) would be different than someone in Macao (Cantonese), and the written standard wasn't actually encoding either of these dialects. So your question should actually be split. How did people learn the language, and how did people learn to read the documents. The spoken language would have happened first, as it almost always does except for in some rare cases.
what about stuff like 混亂 (confusion), 愛 (love), etc. How would these things have been communicated to the foreigners?
The other misconception is that the Chinese writing system is a system of pictures. While 山 originated as a picture of a mountain, by the 1500s it was no longer a picture any more than 愛 is a picture of love. 日 doesn't actually mean "sun" in most cases, despite this origin. In modern Mandarin, 月 by itself more often has the meaning of "month", not moon.
Getting back to the point, there are people today who learn languages without classes and textbooks. This is the way someone would have learned languages back then as well, by picking it up slowly through and spending time with native speakers. However this is all assuming that someone in this possition – an explorer or newly arrived missionary – really did learn the language instead of going through either a translator or another language altogether, either of which would have been readily possible at the time.
Separate out the written language from the spoken languages because they were definitely different things and it will confuse the question you're asking to try to address them together.
(edited for clarity)
I'm not sure how much Macao used the Chinese language at the time.
Assuming you mean "Chinese" as "Mandarin", the answer is not at all. As another commenter has already said, there was a common written form, Classical Chinese, but it's not the same thing that people were speaking, and Mandarin has never been spoken widely in Macau until after it became part of the People's Republic of China in 1999, or before then by it was only by people working in the tourist industry.