Why did Gandhi fail to win the hearts and minds of Muslims in the British India?

This is obviously a massive question, and I hope my answer can do it a little justice if no others materialise. Full disclosure: due to a loss of reading lists my advice on secondary sources is limited but I can point you to some primary ones - For undergrad I did a paper on the history of the Indian Independence Movement 1919-39 and the failure of the INC to avoid division is a key question for the period. I should also state now that this is by its very nature a highly political topic and others may well disagree with the below.

First things first - Gandhi himself was relatively popular even among Muslims (in fact moreso with them than diehard Hindu nationalists who saw him as a traitor, one of whom would of course eventually assassinate him), the question is how did the Indian National Congress manage to fail to prevent demands for a separate Muslim state despite this popularity.

A useful bit of context here is the 'Divide and Rule' theory that often rears its head to this day when the topic is discussed - it is an undeniable fact that communal divisions were useful to the British, and that Britain did not hesitate to make use of them rhetorically to deflect demands for home rule or independence. However, just because Britain often made things worse (and was keen that Hindus and Muslims should not get along too well), this does not mean that Muslim insecurities were simply imagined neuroses.

Hindu-Muslim tensions had been rising throughout the 1920s and communal riots leaving many dead had become commonplace- one calculation is 72 riots in the period 1923-26. It is significant that this came fast on the heels of the first big, nationwide self-rule movement, the Non-cooperation movement, which emphasised resisting British rule through non-violent means.

Paradoxically, the Khilafat Movement, which united Muslims in persuading the British to protect the institution of the Islamic Caliph, formed a large part of that campaign. In the aftermath of WWI and the collapse of the Ottomans, it appeared that this symbolic head of the worldwide Muslim community was to be abolished. Hindus fighting for independence took up this cause as part of their movement, and this co-operation is often seen as a high point in Hindu-Muslim political relations in the period.

Riots often broke out over the killing of cows or the playing of music (for Hindu festivals) outside mosques. The Congress did make attempts to host unity conferences and promote peace, and Gandhi himself of course famously promoted interfaith dialogue and understanding. But it was to no avail; the riots continued essentially unabated and with each one the spark for further violence and discord was set.

The real cause of these tensions is hard to divine and there are many explanations which I don't propose to delve into in great details because I am not hugely familiar - whilst the sparks appear to usually have been religious as above, often there was an economic element; e.g Muslim farmers angry with Hindu moneylenders, or in Bombay in 1929, Hindu workers vs. Muslim strikebreakers.The important point for us here is that violent disturbances were frequent, and they almost always became a Hindu-Muslim conflict.

The government wasted no time in attributing the upswing in communal tensions to the agitations of the Congress, and insinuating that whilst the INC claimed to speak for all of India, it did not truly speak for Muslims.

The INC's answer came in the form of a report on the Cawnpore Riot of 1931, a particularly brutal incident which destroyed the city and left hundreds dead. The British Government report, essentially, lay the blame, of course, at the foot of the infantile Indians who could not be trusted to play nice.

The Report produced by the Congress is a fascinating document - as appended to it is a very long account of the history of Hindu-Muslim relations in India, whose basic thesis is that Hindus and Muslims got along just fine until the British period, and that generally it is 'foreign interference' which is to blame for Indian strife. Again, I won't comment on the accuracy of this potted history, the important point was that it was a convenient story with a clear purpose. One further point to note about this history is that it holds up 'separate electorates' (reserved seats for Muslims on colonial legislative councils which were introduced in the 1909 Government of India Act) as an example of Britain's duplicitous attempts to frustrate and confuse Indian attempts at self-governance.

The Report itself then goes on to imply very carefully that the local government and police force were not just completely negligent in their attempts to put down the riot, but quite possibly pursued a duplicitous policy of 'benign neglect'; they failed to properly restrain the conflict, in order to show local Muslims and business people (Cawnpore was at the time a very important commercial centre) what they could expect if the British left India and sow divisions which would prevent unified independence movements.

The INC's view is clear - Hindu-Muslim tensions are an invention of the British and a result of foreign interference. Indians can and will live together in peace if only they can rid their land of this foreign interloper, and any concessions to communalism are really concessions to the British.

I think it's important to flesh out this narrative because I think it informs how the INC react to demands by Muslim political leaders (in the form of the Muslim League) for specific minority protections for Muslims in India. The INC's purpose, for the most part, is a strong Indian state - even if they recognise the need for Federalism, they have a vision of a unified India with a strong centre and see any concessions protecting Muslims as a separate class would only serve to weaken India and turn British-fanned communalism into a permanent feature of Indian society.

This then sets the scene for the discord over the Nehru report - a blueprint for Indian governance that served as a counterweight to the Simon Report, which was commissioned by the British government and gave very limited proposals for Indian home rule - which marks the true break between the Muslim League and the INC. The Nehru Report rejected demands for reserved electorates for Muslims in legislatures as had been introduced in the 1909 Act, and furthermore, it posited that 'residuary powers' should be reserved to the centre rather than the States (the opposite of the case in, say, the United States, where everything not specifically reserved to the Federal government is the preserve of the States).

This was unacceptable to Muslims - they would be in a minority everywhere except the Northwest and Bengal (the parts that would become East and West Pakistan), and so would be subject to the whims of Hindu voters. The strife of the 1920s showed what an insecure existence this could prove to be. Only strong states and reserved seats at the centre could properly guarantee their position in a united India. This was something that the Hindu-dominated congress simply could not bring itself to agree to - it was the circle that could not be squared.

In summary, the Congress was unable to accede to the political demands of Muslims, who, fearing Hindu dominance, wanted concessions to protect Muslims. For all but a few Muslim leaders, this sealed the case for a separate Muslim state. This was all against the background of typical imperial tactics by Britain, who had no interest in assisting Hindus and Muslims in living together harmoniously.

EDIT: After re-reading my reply, I think I should make clear, the road to the Lahore Declaration was a long one and I am only giving a high-level political account of why the Indian National Congress could not accede to the political demands of the Muslim League. Obviously there were many Muslims in Congress who stayed with the idea of a unified India until the end (e.g Dr. Ansari), and there are further interesting discussions to be had about the reality of Hindu-Muslim relations in India on the ground, not to mention the role that Hindu extremists and the development of Hindutva ideology had to play in this. As I said at the start, the question of partition is a huge one.

What I set out to show was how difficult it was to reconcile Muslim demands with Congress' vision of a united India, and how the climate of debate set by the British made the Congress defensive about even acknowledging the reality of the Muslim question.