To be clear, I'm fairly aware of the immediate causes. In both the South African and Rhodesian cases, the settler class was the primary force behind independence to begin with, whereas, in the case of Kenya, Kenyan nationalists were the driving force.
I'm more interested in the historical factors which caused this divergence in the first place. Why did British and settler efforts to create a settler oligarchy in Kenya fail as opposed to South Africa and Rhodesia? Was it merely a demographic issue, because of the extent to which africans outnumbered settlers in Kenya as opposed to South Africa or Rhodesia?
You are right that demographics did play a part. European Kenyans were only 60,000 people in a colony of 5.5 million in 1960 (about 1%). In contrast, there were ~225,000 White Rhodesians in a population of 3.8 million in 1961 (about 6%).
However, I'll hasten to add that the Kenyan nationalist movement was not monolithic. In the 1950s it was split into the British-educated moderate-conservative wing (the Kenya African Union led by Jomo Kenyatta) and the radical wing (Kenya Land and Freedom Army led by Dedan Kimathi).
A major fault-line separating these factions of the nationalist movement was where they stood on the Land Question.
British settlers began to arrive in Kenya in 1900, and there were two major waves of migration in the 1920s and after World War 2. The initial period of settlement from 1900-1930 brought in Britons who sought to pursue large-scale commercial agriculture (and these settlers saw themselves as British landed gentry transplanted to Kenya). These settlers overwhelmingly chose to live in the central highlands (the area around Nairobi and Mt Kenya) which featured the best agricultural land and a climate free of malaria or sleeping sickness.
White settlement resulted in the large-scale removal of African populations from their historical lands in the central highlands (called the White Highlands) and movement to Native Reserve Areas. Simultaneously, colonial development schemes sought to bring african peasant farmers into the cash economy as well as raise revenue for the colonial administration. The introduction of the Hut Tax, required to be paid in cash, effectively forced both landless peasants as well as those fortunate enough to have some land in the Native Reserves to seek employment on European farms as contract laborers, raising cash crops. White settlers also lobbied for laws to ban African peasants from growing cash crops (e.g. coffee) in Native Reserve Areas, in order to prevent competition as well as ensure a pool of migrant labor that had to work on European farms as one of the few sources of cash in the colonial economy.
However, there were segments at the top of African societies who made gains within this system. People who had familial ties to "customary chiefs" or subordinate political elites could often get access to farm land in the Native Reserve Areas. Being at the top of African society often correlated with being able to send children to mission schools. Education and literacy unlocked opportunities for work as teachers, clerks in the colonial administration, or for railroads or merchant shipping. These were all at the bottom rungs of the administrative or commercial hierarchies, but these jobs provided cash salaries outside of agricultural labor. In exceptional circumstances, some of these mission-educated youth were able to pursue university education at Makerere University in Uganda or even at the University of London. Jomo Kenyatta, the president of the Kenya African Union, was one of the lucky few who studied in London and was exposed to Pan-Africanist and nationalist discourse from fellow black students from Africa and the Caribbean in the 1940s.
One result of the policies of land dispossession and promotion of migrant labor was the creation of a large class of landless african peasants who resented the alienation of their ancestral lands, resented the outright racism they encountered on British farms. These landless peasants also resented the opportunities that the African elites enjoyed within the colonial system like owning land and salaried jobs.
These dispossessed peasants engaged in "squatting", illegally taking over farms, both in the reserves and in the White Highlands throughout the 1910s-1940s. In the late 1940s and early 1950s, the militancy of these squatters increased, and landless peoples from several populations including Meru, Embu, Kalenjin, and major participation from Gikuyu, joined in the Kenya Land and Freedom Army. This organization protested against racial hierarchy and greater White economic opportunity in the colony. Starting in 1952, they began armed attacks on Gikuyu, Meru, Kalenjin loyalists who benefited from the colonial order. The KLFA also began attacking settler farms, although the number of Europeans who were killed or wounded in farm attacks between 1952 and 1960 was under 100. African "loyalists" suffered much worse violence from this armed anti-colonial populist organization.
Because of the attacks on White farmers, the colonial administration became very unnerved at the prospect of armed insurrection in the colony. In 1952 Governor Baring declared a State of Emergency in the colony, and Prime Minister Winston Churchill ordered soldiers from the British army, the Kings African Rifles, and Kenya police to engage in counter-insurgency operations to break the KLFA, which the British press called the Mau Mau. This counter-insurgency effort also sought to create "native-auxiliary police" among the loyalist elements of Gikuyu, Kalenjin, Meru society who were targets of KNLA/Mau Mau attacks.
The British press also engaged in a sustained propaganda effort, widely publicizing lurid tales of farm attacks on White settlers and portraying the Mau Mau as a savage, primitive cult aimed at wiping out the White population in the colony.
The Mau Mau Uprising lasted from 1952-1960, but it's most intense phase was from 1952-1956, which ended when KNLA leaders Stanley Mathenge and Dedan Kimathi were captured and executed in 1954 and 1956, respectively.
Counter-insurgency operations included the villagization of Gikuyu, Meru, Emba and Kalenjin populations to separate them from KNLA cadres who were hiding in forest reserve areas. It also included placing captured KNLA militants into detention camps where they were routinely tortured to extract information about location of KNLA/Mau Mau camps in the forests. Also, it meant the use of RAF aircraft dropping bombs against KNLA camps, particularly in the forests around Mt Kenya.
As the counter-insurgency operations dragged on in the late 1950s, Britain was facing a changing situation in the empire. Sudan declared independence from Britain and Egypt in 1956, Ghana gained her independence in 1957, and Nigeria was on the path to gaining independence in 1960. In 1960 British Prime Minister Harold MacMillan would declare in a speech in Accra, and again weeks later in Cape Town:
The wind of change is blowing through this continent, and whether we like it or not, this growth of national consciousness is a political fact. We must all accept it as a fact, and our national policies must take account of it.
....
As I have said, the growth of national consciousness in Africa is a political fact, and we must accept it as such. That means, I would judge, that we've got to come to terms with it. I sincerely believe that if we cannot do so we may imperil the precarious balance between the East and West on which the peace of the world depends.
The world today is divided into three main groups. First there are what we call the Western Powers. You in South Africa and we in Britain belong to this group, together with our friends and allies in other parts of the Commonwealth. In the United States of America and in Europe we call it the Free World. Secondly there are the Communists – Russia and her satellites in Europe and China whose population will rise by the end of the next ten years to the staggering total of 800 million. Thirdly, there are those parts of the world whose people are at present uncommitted either to Communism or to our Western ideas. In this context we think first of Asia and then of Africa. As I see it the great issue in this second half of the twentieth century is whether the uncommitted peoples of Asia and Africa will swing to the East or to the West. Will they be drawn into the Communist camp? Or will the great experiments in self-government that are now being made in Asia and Africa, especially within the Commonwealth, prove so successful, and by their example so compelling, that the balance will come down in favour of freedom and order and justice? The struggle is joined, and it is a struggle for the minds of men. What is now on trial is much more than our military strength or our diplomatic and administrative skill. It is our way of life. The uncommitted nations want to see before they choose.
This speech reflects PM MacMillan and the Conservative party's growing acceptance that independence was a fait accompli in Africa, that the example of Sudan, Ghana and Nigeria (as well as ex-French, ex-Belgian and ex-Italian colonies in gaining independence in 1960) were showing that African peoples could govern themselves as independent countries, which would only increase demands from Uganda, Kenya, Tanganyika, Zanzibar, Nyasaland, North Rhodesia, South Rhodesia and South Africa for independence and black majority rule.
So, it was the position of the MacMillan government that Kenya colony would need to grant concessions to the "moderate nationalists" of the Kenya African Union, to reward loyalist support during the Mau Mau uprising and outflank the radical positions of the KNLA. So, from 1956-1960 the colonial administration gave concessions allowing secure land tenure in Native Reserves, allowing growing cash crops, allowing African land ownership in the White Highlands, and increasing the number of seats in Kenya colonial parliament representing African populations. That last concession is key, because colonial Kenya's parliament operated in a system of "racial colleges" where a set number of seats were reserved to represent the white population, a set of seats to represent the South Asian population, and another set of seats were reserved to represent the African population. The concessions in the 1960 Lancaster House agreement increased the number of seats assigned for African politicians so that they outnumbered White and Asian seats. 1/2