EDIT: This is an April Fools' Day post! Confucius was NOT an arrogant egotist, the literati were NOT useless for hundreds of years of history, and "Young Confucianism" is not a thing.

Great question! Very generally speaking, Neo-Confucianism is a social and ethical philosophy, which is based on Confucianism. Even though the founding ideas of Neo-Confucianism came from some Tang dynasty Confucian philosophers (notably Han Yu), the ideology did not gain any prominence until the Song dynasty, and even then it would not be until the Southern Song that Neo-Confucianism really began to proliferate, and become entrenched within the mindset of the elite class - the literati, or wenren (文人).

Anyway, The Analects (论语) is probably the best place to start in understanding Confucianism and its variants. The Analects, as you may already be aware, is a collection of the sayings and ideas associated with Confucius. In the Han dynasty, The Analects was only ever considered to be a commentary on how to interpret other, more important works (the 5 classics), but over time, The Analects came to be recognised as one of the four key, foundational texts of Confucianism. People often consider The Analects as the most important philosophical text in Chinese history, though the reality is that the text was often used to teach youths, before moving on to the 5 Classics, which were considered more important. It was only in the Song dynasty that The Analects began to be considered an essential text.

In the Song dynasty, the examination system became increasingly used as a source for hiring officials. Since The Analects was now being considered of greater importance than before, examination candidates would study The Analects in great detail, often dedicating years to decades of their lives attempting to learn the words of Confucius, for the purpose of passing their examinations, and entering the ranks of the civil service bureaucratic elite.

Unlike previous dynasties, in which the upper class more closely resembled an aristocracy, the examination system in the Song redefined the upper class as scholars, who had earned their position in the elite as a result of their intelligence, and their knowledge of the core Confucian texts. Confucianism came to be associated with the idea of a stable and successful society, with the literati at its head. This viewpoint meant that the literati viewed any society without Confucianism as inferior. This idea stems from the idea of siwen (斯文), translated by Peter Bol as “This Culture of Ours,” and it created an us v. them dichotomy in which the Song dynasty was the superior, and all of the surrounding “barbarians” were the inferiors.

Unfortunately for the Northern Song dynasty, one of the surrounding barbarian groups, the Jurchen, was not inferior - at least, militarily speaking. They invaded the Song state, taking the capital city, and absorbing somewhere between 1/4th and 1/3rd of the population of the Song into their Jin dynasty. The Song state, meanwhile, re-established itself as a smaller state in the South called the Southern Song.

The literati were in complete shock. They simply couldn’t come to terms with the fact that their “superior” society had been so easily overwhelmed by the inferior barbarians. “This Culture of Ours” had not lived up to its name. This caused many literati to start questioning the efficacy of their worldview. The literati wondered, if they had been less arrogant and conceited, would they have recognised the Jurchen threat in time to stop it? If they had placed less faith in the notion of “The culture of ours” as the absolute ideal, would they have been less blind to the imminent threat? And, more importantly, how could the Southern Song recover, fight back, and reclaim their lost territory?

It is under the context of this catalyst that Neo-Confucianism truly began to grow. It offered an alternative way of viewing the Confucian classics, which explained not only how the Jurchen was able to defeat the Song, but also offered a solution to the pressing problem of the Jurchen. This solution comes primarily from a re-examination of one of the most famous lines in The Analects, which the literati held to be of great importance:

子曰：「吾十有五而志于學，三十而立，四十而不惑，五十而知天命，六十而耳順，七十而從心所欲，不踰矩。」 The Master said, "At fifteen, I had my mind bent on learning. At thirty, I stood firm. At forty, I had no doubts. At fifty, I knew the decrees of Heaven. At sixty, my ear was an obedient organ for the reception of truth. At seventy, I could follow what my heart desired, without transgressing what was right.” (Analects 2.4)

The literati had been striving to emulate the sageliness of Confucius for hundreds of years, and this passage provided a rough map of how Confucius himself became a sage. We see him go from a youth, eager to learn, to a confident adult in his 30s who “stood firm.” However, by age 40, Confucius started to become a bit overconfident in his abilities, “having no doubts,” and as he aged, he seemed to become more certain of the fact that his actions were always the correct actions. This is not unheard of; even today, one constantly hears the mantra of how the elderly know better than the youth. It ultimately creates a sense of superiority and invulnerability, which can be seen any time an elderly man hobbles across the street with his cane without waiting for the light to change, or any time an old woman insists upon the validity of her coupon that had expired months before. This exact same sense of entitlement could be seen in the literati who took Confucius as their model, and it created an air of invulnerability which prevented the Northern Song from viewing any of the non-Confucian “barbarians” as a threat of any kind.

However, Neo-Confucianism was not willing to abandon Confucius entirely. The Analects, after all, were filled with sagely words of wisdom, and had benefited society up until this point. Neo-Confucians still viewed Confucius as a sage, who from his youth had understood the Dao (the way), yet the turbulence of the Warring States period in which he lived had caused Confucius to adapt, to stray slowly but surely from the purity of the Sage, for the benefit of himself and his followers. Thus, Neo-Confucians decided that they would still follow the words of Confucius, but only the words he uttered as a youth, or as a 30-year-old adult. This distinction was made by judging the arrogance in Confucius’ voice, and is best explained by comparing two passages from The Analects:

子曰：「文，莫吾猶人也。躬行君子，則吾未之有得。」 The Master said, "In letters I am perhaps equal to other men, but the character of the superior man, carrying out in his conduct what he professes, is what I have not yet attained to.” (Analects 7.33)

冉子退朝。子曰：「何晏也？」對曰：「有政。」子曰：「其事也。如有政，雖不吾以，吾其與聞之。」 The disciple Ran returning from the court, the Master said to him, "How are you so late?" He replied, "We had government business." The Master said, "It must have been family affairs. If there had been government business, though I am not now in office, I should have been consulted about it.” (Analects 13.14)

In the first excerpt, we see Confucius still in pursuit of the qualities of a “superior man.” He admits his abilities when it comes to writing, but is still modest enough to recognise fields in which he must improve. Compare this to the haughty arrogance of the second passage, in which Confucius assumes that his disciple is lying to him, based on the premise that he would be consulted if any government business should occur. The first excerpt sounds much more like the Young Confucius as described in Analects 2.4, while the second sounds much more like the Old Confucius.

Neo-Confucianism, then, was an intellectual effort to discern which aspects of Confucianism were still worth following, and which aspects ought to be discarded. The decision to focus more on Confucius’ youthful sayings and ideas is reflected in the English term: “Neo” from the ancient Greek νέος meaning not only “new,” but also “young.”

However, it is easy to see how people might not necessarily agree upon which passages of the Analects come from a Younger, versus an Older Confucius. One of the more popularly contested sections of the Analects is from 5.4:

子貢問曰：「賜也何如？」子曰：「女器也。」曰：「何器也？」曰：「瑚璉也。」 Zi Gong asked, "What do you say of me, Ci!" The Master said, "You are a utensil." "What utensil?" "A gemmed sacrificial utensil.”

On the one hand, some Neo-Confucians chose to view this passage as an Old Confucius passage, since Zi Gong was a hard-working civil-servant who had still not managed to attain the Confucian ideal. Despite his attempts to help the people, Confucius still refers to him as a tool, for his inability to follow the Confucian way. On the other hand, some Neo-Confucians insist that calling Zi Gong a gem-studded tool is something that only an immature Confucius would have said, and thus maintain that the passage still has merits.

I won’t go into too much more detail about the different passages, but if you want to look them up on your own, here are some of the passages considered to be Old Confucius: 10.8, 10.15, 5.27, 18.4, and 7.37. And here are a few passages considered to be Young Confucius: 9.10, 2.12 (debatable), 5.5, 5.10, 11.1, 14.30, and 11.12 (also debatable). All of these passages are on ctext.org, and that is where I have copied the English translations from.

Anyway, the uncertainty involved in determining which passages to study and which to eschew led to the rise of several different schools of thought within Neo-Confucianism, the two most prominent being Lixue (School of Principle) and xinxue (School of the Heart-Mind.) The leaders of these respective schools were Zhu Xi and Lu Jiuyuan.

This is Part One of a two part answer - I'll post Part Two in a bit, and talk some more about Zhu Xi and Lu Jiuyuan. Stay tuned, and feel free to ask questions in the meantime!