To quote Jeaneane D. Fowler: "It is impossible to say exactly why and how all the castes came about."
In very simplistic terms, the Indian caste system comprises jatis (hereditary groups), which are cateogrized under four varnas (classes). There are also outcastes or "untouchables", who fall outside the varna system.
Varna
The varna system is of Vedic (Indo-Aryan) origin, most probably derived from the trifunctional class system (priests, warriors and herder-cultivators) of the proto-Indo-European society. The Iranian texts also mention a similar class system comprising of priests, warriors, cattle breeders and artisans.
Jati
We don't know for certain how the modern jatis originated, when did they become strictly hereditary, when they became the primary defining identity of a person, or when they became endogamous groups.
Genetic research suggests that the majority of the Indian population is an admixture of two post-neolithic groups: Ancestral North Indians (ANI) and Ancestral South Indians (ASI). These are roughly analogous to proto-Indo-Aryans and proto-Dravidians. (Note: the modern Indo-Aryans and Dravidians are linguistic groups: a lot of Dravidians have ANI genes and a lot of Indo-Aryans have ASI genes).
This genetic study suggests that social stratification was already established in Tamil Nadu (a South Indian state) 4000-6000 years ago, much before the Vedic period or the ANI-ASI miscegenation. These castes or classes might have derived from the social hierarchy resulting from the emergence of agricultural technology.
This genetic study suggests that the ANI-ASI miscegenation started ~4000 years back. However, around ~2000 years ago, various endogamous groups were established. Manusmriti, which describes an elaborate social stratification system, was also written around ~2000 years back. So, some of the jatis may have been established around this time. The concept of "untouchables" probably came into being around this time, and was used to discriminate against the groups that were not part of the post-Vedic society or against people whose occupations were 'impure' or unhygienic ( e.g. tanning, dealing with dead bodies, sweeping etc.)
And there are well-documented cases of caste identity formation after this period (e.g. "Jats" and "Rajputs" in medieval India).
All we can say is that different social groups in different regions originated at different times, for different reasons (social, economic and/or political). Many jatis are occupational. For example, Kumhar is the jati of potters and Mochi is the jati of shoemakers. So, one theory is that some of the jatis may have derived from ancient occupational guilds, which over time, became hereditary, as children inherited occupations from their parents. However, many jatis are also based on kinship (tribes and clans) and region.
The British introduced the modern census to India, which might have made the caste system more rigid. The obsession to categorize every jati under one of the varnas seems to be a relatively recent (British-era) development. Many caste identities first became prominent during the British period (e.g. "Yadav" and "Bhumihar"). Therefore, many modern writers blame the British for "creating" the caste system as we know it today. However, this is over-simplification: we do know that even before the British period, the concepts of caste "purity" and endogamy were not alien to the Indian masses.
Sources:
Edit: Fixed links