I've seen some people mention that it was created for the good of society ages ago but how true is that really? And if it really was created for the good of society, then why and when did it change so drastically to be a oppressive force to oppress the people for thousands of years?
The manner in which most scriptures define caste, the caste system or its development and evolution, has little to do with how castes and caste hierarchies actually developed in the real world. For example, texts such as the Purusha Shukta may describe how the "cosmic being was divided into 4 parts and out of each part namely the head, the arms, the torso and the legs, each of the varnas were formed namely the Brahmins, the Kshatriyas, the Vaishyas and the Shudras respectively". However this implies two things, 1) That the caste an individual was born into dictated the occupations he was allowed to choose in life and that throughout history these were the only occupations said individuals chose. 2) That there was a religious justification and foundation for caste first and its real life implications second. However this isn't true. The distinction between the upper or elites in Vedic society and those who would serve as servants or slaves was made in the Rig Veda itself. The Indo-Aryans referred to themselves as Arya or Aryans while they referred to the native inhabitants of the subcontinent as "dasas" literally meaning servants. This relationship between ruler and ruled and strict class and race based distinctions were not unique to Indo-Aryan society. Now, while there was almost a millenia during which heavily male dominated groups of Indo-Aryans took native female partners as well, the idea of the purity of the patrilineal bloodline existed. Meaning while it was acceptable for a elite male to take a dasa female partner the opposite could not be acceptable. It was this patrilineal bloodline that is the foundation of what is known as gotra. A person's gotra is their patrilineal bloodline. Only the three "upper castes" are assigned or have gotras. The Shudras do not. In later centuries the acceptable form of marriage in society would be called "Anuloma" marriages and apart from marriages within the same caste which were considered excellent, this would be the only form of marriage deemed at least acceptable.
With the passage of time, the Indo-Aryan or Vedic society divided itself into 3 upper and 1 lower varna. Namely Brahmin, Kshatriyas, Vaishyas and Shudras. The first three being born out of the fact that occupations and their know how was passed from fathers to sons and hence occupations themselves became hereditary. Overtime, religious justifications for this hereditary system were created. There is also the idea that one could fluctuate between varnas, which is also unfounded in history.
So, caste, or Jati is as old as 2000 years give or take, and the foundation for this system of differentiation between groups of people was the perception of the Indo-Aryans towards the natives of the lands they came to settle upon and the relationship between a ruling elite and it's servant class/ethnic group.
The medieval period in Indian historiography, usually dated between the 12th century onwards to the beginning of the 18th century was a period that saw the emergence and developnent of new institutions and social classes owing to the advent of Turks, Afghans, Iranians and Central Asians as a result of the establishment of several Islamic kingdoms in North India and subsequently throughout the entire subcontinent, since the beginning of the 13th century.
India by this time had well developed and defined caste structures throughout the subcontinent. This system divided society into 4 major divisions, namely, Brahmins, Kshatriyas, Vaishyas and Shudras. The Brahmins occupied the highest position in the caste structure, the Kshatriyas the second the Vaishyas the third and the Shudras the last and fourth place within the caste stricture. Those who chose to remain outside this structure were considered "Mlechha" that can mean both "unclean" or "untouchable". According to religious scriptures such as the Purusha Shukta, The Manusmriti or the Puranas and Upanishads, the duties and responsibilities of each caste were pre-defined. Brahmins were expected to perform priestly functions, apart from this they could also participate in the professions of an educator, a farmer or in times of dire need, could take up mercenary work. The Kshatriyas were meant to be a caste whose purpose was to make war, that is, they were expected to be soldiers. The Vaishyas were expected to be traders and craftsmen while the Shudras were expected to engage in purely manual work, be it farm labour, wage labour etc.
Ofcourse, religious scriptures contradicted each other quite often, and while the usual duties of a Brahmin were priestly work and education, we find instances of Brahmins engaging in military service in the epics and even in the Rig Veda, and even becoming kings in the Classical Period. Thus, in real life, practicality and needs took precedent over scripture. Similarly we find instances of Kshatriyas becoming sages, and following the path of a hermit or sage, a path considered to be the domain of the priestly class. Therefore, contrary to scripture, it was not uncommon to find Brahmins engaged in military work. However, such exceptions, were excluisve to the upper echelons of the caste structure, meaning while Brahmin soldiers could be a common sight, Shudra land owners were not.
By the 11th century, hierarchical structures centred around ownership of land, caste ties and hierarchies, were well established in the Northern part of the subcontinent. The demographic distribution of society during this period is interesting and in some ways predictable. The rural land owning peasantry was formed usually by either the Brahmin or Kshatriya castes, although there were also ambiguous groups, that although didn't fit into any particular caste group in the sense of how castes emerged, still became integrated into Hindu society by the 15th century, as an agricultural group that was often employed as mercenaries. Since there was little to no concept of "individual private property" yet, the rights to one's land were established by recognition of patrimony. Since the Brahmins and Kshatriyas occupied the two highest caste tiers and were the backbone of feudal armies and beaurocratic structures, the granting of land to these groups in return for fixed revenue or military service or both, was common practice. We find instances of grants to Brahmins being made during the Ancient and classical period as well for administrative and political purposes. Such grants were known as "Brahmadeya" literally meaning "given to Brahmins" and were meant to expand agriculture and integrate tribal people's into the caste structure through a process called Sanskritisation.
These land owning peasants provided military service and revenue to the Imperial treasury and these revenues were remitted from these peasants to the Imperial revenue collectors via intermediaries, who occupied a special role in Indian society since the classical or Gupta Era. These intermediaries claimed the ancestral right to collect and transfer revenues and formed an important component of Indian society. While earlier they were known by many names depending on region and language, during the medieval period and especially after the establishment of Mughal rule in the late 16th century, these intermediaries came to be known as zamindars.
Meanwhile, the urban centres were usually occupied by the traders or craftsmen, that is, the Vaishyas. The Shudras on the other hand served as either landless farm labourers in the rural country or as wage labourers in the cities.
Social and economic mobility during the Pre-Medieval period, was usually a feature of the upper echelons of the caste hierarchy, meaning the Kshatriyas and Brahmins. Brahmin and Kshatriya peasants could serve in the levies and armies of their kings and/or "zamindars" or local Rajas and through their continued service they earned the patronage, favour and support of their employer. These systems of patronage were very much centred around caste and clan loyalties. It was common for Brahmin zamindars to rise to prominence in regions where there were many Brahmins to begin with. Such zamindars or local Lords then depended on the continued service and support of his caste brethren to rise to higher offices, meanwhile his supporters counted on continued patronage and opportunities as a reward for said support. This meant opportunities in the beaurocracy and military and that further augmented one's social status and personal wealth. Vaishyas could acquire wealth by working in their respective positions but the social position of Vaishyas was considered to be in service to the Brahmins and Kshatriyas "who maintained the caste structure" as providers of revenue. Meanwhile, the Shudras and the untouchables lacked any if at all means to acquire higher economic or social status.
Now, we look at the medieval period once again. We find that with the advent of the Turks, the Afghans and the Mughals the pre established caste structures remained largely unchanged. While newer beaurocratic roles and positions opened up, the Hindu population of North India, especially its and land owning classes, found themselves competing for royal favour with their Muslim counterparts. This period also saw conversion either voluntarily or/and even forced, of many Hindus from across the caste spectrum to Islam. These individuals, carried their castes with them. As a result, Indian Muslims had a caste structure as well.
Spurces :
"A History of Ancient and Early Medieval India" by Upinder Singh
"India's Ancient Past" by RS Sharma
"Early Medieval Indian Society" by Ram Saran Sharma
"A History of Medieval India" by Satish Chandra