? They did. From as early as 1940 the British, along with the Australians, New Zealanders, South Africans and others, and later the Americans, fought them across much of North Africa. Eventually the Western Allies pushed the Italians and Germans out of there - including a couple of the most famous battles of the war, like Tobruk and Second El Alamein - and launched their invasion of Italy from there.

See u/rough_rider7’s and u/the_direful_spring’s answers here.

Otherwise it wasn’t easy or worth it for the Nazis to traipse across the Sahara to other parts of Africa, nor feasible to send shipping to conquer colonies further south given Allied naval superiority in the Atlantic. France also had colonies in much of Africa, some of which declared for Vichy France, but the Western Allies toppled these for the Free French. For more details there see u/Aleksx000’s answer here.