Genetic studies can shed light not necessarily on when castes came into being but when caste endogamy became the norm. It seems that the advent of caste endogamy differed depending on the particular South Asian population. In any case, it was not a recent thing. To quote from the Discussion section of a 2016 PNAS paper (http://www.pnas.org/content/113/6/1594) :

We have provided evidence that gene flow ended abruptly with the defining imposition of some social values and norms. The reign of the ardent Hindu Gupta rulers, known as the age of Vedic Brahminism, was marked by strictures laid down in Dharmaśāstra—the ancient compendium of moral laws and principles for religious duty and righteous conduct to be followed by a Hindu—and enforced through the powerful state machinery of a developing political economy (15). These strictures and enforcements resulted in a shift to endogamy. The evidence of more recent admixture among the Maratha (MRT) is in agreement with the known history of the post-Gupta Chalukya (543–753 CE) and the Rashtrakuta empires (753–982 CE) of western India, which established a clan of warriors (Kshatriyas) drawn from the local peasantry (15). In eastern and northeastern India, populations such as the West Bengal Brahmins (WBR) and the TB populations continued to admix until the emergence of the Buddhist Pala dynasty during the 8th to 12th centuries CE. "

Earlier, with regards to an AJHG 2013 paper (https://www.cell.com/ajhg/fulltext/S0002-9297(13)00324-8) he coauthored, Harvard Medical School professor of genetics David Reich commented (https://www.boston.com/news/science/2013/08/09/genetic-study-finds-caste-system-in-ancient-india-began-about-2000-years-ago):

"From genetic data, remarkably, you see this picture emerging of cultural change,’’ Reich said. The population mixture didn’t happen in pockets — it was a profound mixing that has left traces in the DNA of people in all areas of India today. But that came to an abrupt halt around 2,000 years ago, likely due to the implementation of the caste system, Reich said.

So the earlier paper seems to put the advent of caste endogamy at about 2000 years ago while for the newer one it seems to start at about 1500 years ago.

To add to u/roelm2's answer, while we cannot necessarily know about when the caste system was established from genetics, it gives us clues as to when it became a vital part of society. From that point of view, genetics are one of our best resources in establishing a chronology, however, outside evidence is needed to contextualize it. For this, we look to texts and epigraphy.

#The Vedas

As far as texts go, we have a somewhat limited view of the early history of caste. Since we cannot definitively date everything and our earliest manuscripts from India are comparatively recent, we have to rely on textual criticism and philology to date them, leading to a wide dating for the origin of Caste. The earliest text we have that mentions the varnas is the Purusa Sūkta, best known as the "Hymn of the Cosmic Man*, from the Ṛgveda.

His mouth Became the Brahmin, from his arms the Rajanya. His thighs became the Vaiśya and his feet the Śūdra.^1

However, this is a later hymn from Mandala 10, and furthermore, seems to be a later interpolation, given that it mentions the three Vedas and the caste system is not attested elsewhere in the Vedas. However, the archaic term for the Kṣatriya attests a somewhat earlier date.

#Upaniṣads

The next stratum of literature we can examine is the Upaniṣads, which are contemporary with the final stratum of Vedic literature and also, in a sense, a continuation of the Vedas. This puts them between the 8th century BCE and the 6th century CE which began the Purāṇic period of literature (although that started as early as the 3rd century CE). However, the principal and most important Upaniṣads were composed between the 8th century CE and the 3rd century CE. Discussion of the caste system, or rather the varnas is seen in depth here.

The Bṛhadāraṇyaka (Sanskrit: Great Forest) Upaniṣad was probably the earliest outside of the Vedas to mention varna as a social phenomenon and also offers a mild critique of some of the social norms of Indian social and philosophical structure. For instance, the principal sage in the work, Yājñavalkya, discusses philosophy with his wife as a near equal, which was quite taboo in those days.^2 On the subject of caste it has this to say;

Verily, in the beginning of this world was Brahma, one only. Being one, he was not developed. He created still further a superior form, the Kṣatrahood, even those who are Kṣatras among the gods...Therefore there is nothing higher than Kṣatra. Therefore ath the Rājasūya^a ceremony the Brahman sits below the Kṣatriya. Upon Kṣatrahood alone does he confer his honor. This same thing, namely Brahma^b, is the Source of Kṣatrahood. Even if a king attains supremacy, he rests upon Brahma as his source. So whoever injures a Brahman attacks his own source. He fares worse in proportion as he injures one who is better.

...He (Brahma) was not yet developed, he created the Vaiśya... He was not yet developed. He created the Śudra varna... He was not yet developed. He created still a further better form, dharma. This is the power of the Kṣatriya.

We also see here the first mentions of outcastes who would later be known as the Dalits or the untouchables in the form of Caṇḍāla's and Paulkasas. While these mentions are brief and insubstantial, they do attest to the developing notions of those outside the varna system.

...So that Brahma appeared as Kṣatriya, Vaiśya, and Śudra. So among the gods Brahma appeared by means of Agni, among men as a Brahman...^3

Here we see the foundations of the later caste system. It is interesting that the Kṣatriya are portrayed here as "lower" than the Brahman on first glance. However, a deeper reading of the texts reveals the nature of the relationship is quite different. The Kṣatriya are seen as the temporal holders of power, the upholders of the dharma but are subservient to the Brahman who are the embodiment of the universal Brahma and thus dharma itself. The temporal control of the Kṣatriya is only a convenience in the sense that it is the duty and purpose of the Kṣatriya to rule, but the duty and purpose of the Brahman to lead in higher matters. For a Kṣatriya to harm a Brahman means that he has gone against his duty and attacked his better, thus the traditional (or perhaps less so) order is upheld despite the dominance of the Kṣatriya rulers on the surface.

To further this, the story of Janaka is relayed in the 4th part of the text. Janaka struggles for an escape from saṃsāra and wishes to know the true nature of ātman (Sanskrit: Self/Soul) and Brahma. Yājñavalkya goes in with the intention of not discussing this with the Janaka, since he is Kṣatriya, however Janaka had previously proved himself worthy by defeating Brahmins in philosophical dialogue and had been given Brahminic status. Furthermore, his direct questions catch Yājñavalkya off guard and he accedes to Janka's requests to these answers. Eventually, even these roles are subverted as Janaka offers to be Yājñavalkya's slave and gives him his wealth in gratitude.^4

The Chāndogya (Sanskrit: from the root Chanda- meaning poetic) Upaniṣad began composition slightly after the Bṛadāraṇyaka Upaniṣad and ended either around the same time or slightly later. It is perhaps the most studied Upaniṣad and goes deeper into discussions of Brahman, but also presents more social teachings.

Perhaps the most famous story from the Chāndogya is from the 4th section, and details the story of Satyakāma Jābāla who wants to learn the Vedas, but does not know his varna since his mother had too many partners to know which his father belonged too.

One day Satyakāma Jābāla said to his mother Jabālā: "Mother, I want to become a vedic student. So tell me what my lineage is. She replied "Son, I don't know what your lineage is. I was young when I had you. I was a maid then and had a lot of relationships. As such, it is impossible for me to say what your lineage is. But my name is Jabālā and your name is Satyakāma. So you should simply say that you are Satyakāma Jālbāla.^c"

He went to Hāridrumata Gautama then and said: "Sir I want to live under you as a vedic student. I come to you, sir, as your student."

Hāridrumata asked him: "Son what is your lineage?" and he replied, [the thing his mother told him]"

Hāridrumata then told him: "Who but a Brahmin could speak like that! Fetch some firewood, son. I will perform your initiation. You have not strayed from the truth."^5

Thus we see a young man that would typically be considered an outcaste elevated through his knowledge and perseverance. This shows that there was a certain fluidity in ancient Indian society. We also see a changing definition of who a Brahmin is, from one who is simply born as one, to one who has the proper knowledge.

Earlier in that same section, we see the tale of a Brahmin who lives in poverty and has a noticeable skin condition. A king approaches him and begs for his instruction, but he rejects him, calling him a Śudra (who weren't allowed to hear the Vedas, let alone learn them and their meaning). The king makes greater and greater offers until he offers his daughter, which the Brahmin accepts. This shows the emphasis placed on knowledge over position and that the Varna system, at least ideally to the writers, was very still hierarchical.

We also see here that the first mention of extreme impurity for the outcastes:

Now, people here whose behavior is pleasant can expect to enter a pleasant womb, like that of a woman of the Brahmin, the Kṣatriya, or the Vaiśya. But people of foul behavior can expect to enter a foul womb, like that of a dog, a pig, or an outcaste woman.^6

We see a comparison here of outcastes to animals, and not just any animal but dogs and pigs who were seen as ritually impure. The radical contrast here illustrates the way that outcaste people were seen by the Brahmins and the orthodox society and how disliked they were, however, it should be noted that society had not developed the complex bureaucracy and social structure required to enforce these social ideals.

#Dharmaśastras/Dharmasūtras

We must also look to the Dharmasūtras, which were composed between 400 BCE and 100 CE.^7 The most notable of which are the Āpastamba and the Gautama. These Dharmasūtras were ideal law and ethical codes, like the more famous Manusmṛti. Whether they were ever actually enforced is a matter of some debate, but they were composed by high class brahmins.^8

The first among these, the Āpastamba was likely composed in the 4th century and begins with an explanation of the social classes:

There are four classes: Brahmin, Kṣatriya, Vaiśya, and Śūdra. Among these each preceding class is superior by birth to each subsequent. Those who are not Śūdras and ar enot guilty of evil deeds may undergo initiation, undertake vedic study, and set up the sacred fires: and their rites bear fruit. Śūdras are to serve the other classes, the higher the class they serve, the greater their prosperity.^8

The further instructions are distinct for each social class. We see a rigid structure beginning to form here, with the varna structure systematized in a way that it hadn't been before. We also see the more rigid structure clearly presented. Again though, this does not mean it was followed.

Notes

a: An archaic form of Kṣatriya, Rajanya was likely the name for tribal chiefs, rather than the more powerful kings and nobles.

b Brahma here is not just the "god" but also the idea, we see here the predecessor to the later Upaniṣadic Brahman.

c A Vedic saint's name.