The British lost some battles, most notably at Isandlwana, but they (a) won the war, and (b) the war was mostly fought in Zululand (Rorke's Drift was on the Natal-Zululand border).
The war was generally not a problem for British (or Boer) settlers. Many Zulus suffered immensely, since the British forces stole ("confiscated") cattle and other livestock, burned crops, and burned villages. This was a deliberate attack on the economic foundations of Zulu society, and caused much hardship and even starvation. After the initial British losses in the first invasion of Zululand, there were calls for genocide of the entire Zulu nation - the actual prosecution of the war stopped well short of genocide, but was still thoroughly brutal. Apart from the deliberate attacks on the Zulu civilian economy, there was widespread killing of Zulu wounded, and killing of Zulus attempting to surrender.
For more on this, see