It's hard to say, especially because much of "the truth" has been obscured by people with vested interests.

Sarila's book, reviewed here, suggests that the British cut a deal with Jinnah so that he would get Pakistan, and in return support British and American interests militarily. The idea was to have a staunchly pro-Western and loyal base close to the oil kingdoms, to counter Soviet influence.

Nehru was known to have socialist leanings, and be pro-Soviet Union.

One of the mysteries about partition is the advancement of the date of independence by a year - a decision that Mountbatten took with undue haste. The Hindi novelist Kamleshawar, in his epic Kitne Pakistan suggests that the date was advanced because the British government knew Jinnah was terminally ill, and there would not be a Pakistan without him.

Stanley Wolpert's Shameful Flight is scathing in its treatment of Mountbatten's actions during his tenure as the last viceroy.

I think definite answers will be hard to come by, but for sure it had to do with the personalities of a handful of leaders, and the schemes of the British Raj.