I've written on this topic before, so I will just give you the answer I usually give. If you have any specific questions, don't hesitate to ask:

Counterinsurgency conflicts are not won entirely by military means. By treating it is as a conventional military conflict, which is very common for a layman when reading about a subject like the Vietnam War, you are doing a disservice to yourself. In counterinsurgency, those that effectively combine social and military means into one strategy are those that win.

The Rhodesians number one priority was the elimination of the guerrillas. This is known as a direct approach, but which means that all the other concepts that are so important, such as winning the hearts and minds of the population etc., are given a second rate role. While there were within the armed forces classes on 'African customs' given, the truth is that an understanding was never properly reached and that gap between the white Rhodesians and the black Rhodesian was always present. The Rhodesian Light Infantry was very qualified in its military role. It was trained for COIN missions, it knew exactly what it was doing and the Rhodesian military had great success in military terms with its infamous Fire Force tactics. However, not even that could control the fact that the insurgents constantly grew in size and would soon enough stretch the capability of the Fire Force to deal with all the incidents of guerrilla warfare. The insurgents themselves learned soon enough to adopt tactics to evade Fire Force, something that is conveniently overlooked in much of the 'military history buff' literature out there. In the end, the Rhodesian way of warfare was seen as innovative and hailed as being effective by contemporary voices. They were true military professionals, but to quote Paul Moorcraft: "But the 'field' in revolutionary warfare is not the same as that in conventional warfare. In a guerrilla war the battlefield is the political loyalty of the mass of the population. The Rhodesians did not develop tactics to win enough battles in that more subtle war."

More or less, you do not win a counterinsurgency war by being brutal. No matter how many battles and skirmishes you win against insurgents, your brutality against the civilian population on which you and your enemies support depend on are going to turn all of them to the other team. The Rhodesians were not the first to commit this mistake and definitely not the last.

Some add-on reading regarding the Bush War from other users:

/u/plusroyaliste on the results of the conflict out of a political standpoint.

/u/Profrhodes killing it as usual in What was white Rhodesia's 'endgame' during the Bush War? and in this thread on segregation and black soldiers in the Rhodesian armed forces.