Most African borders are indeed drawn by colonial rulers without much regard for existing political structures; the colonisation was a process with many competing interests, both between countries and within them (e.g. companies, colonial authorities), and thus as with any 'land acquisition' there needed to be some way to divide the territories and administer them efficiently. The nature of the boundary lines chosen varies, generally if a natural barrier like a river or mountain range was available, this would be used, and if it wasn't then a straight geographical line would be chosen. This is quite similar to how the borders of US states were established.

European settlement began on the coast in the 15th - 18th centuries, so one type of border, especially in west Africa, reflects European control of a coastal strip that was then extended inland in the 19th century either to a river (e.g. Ivory Coast) or to a certain latitude (e.g. Ghana).

This can also be seen in Mozambique, where for hundreds of years Portuguese control was limited to the coast (EDIT: see /u/Joltie 's correction below), but during the scramble for Africa claims to the inland for formalised, and control was extended. Here the boundaries were rivers (Lugenda with Tanzania), mountains (with Zimbabwe) and some straight geographical lines when no natural barriers were available. Where the borders would fall was a result of compromise with other European contries, in this case with Britain. Portugal originally aimed to control all the land between Angola and Moambique.

Many other colonial African borders are also rivers, for example if you zoom in on any irregular border of an African countries in google maps, you'll often find a river there. Some examples are large parts of the border between Botswana and South Africa (Limpopo), Zimbabwe and South Africa (Limpopo), Zimbabwe and Zambia (Zambezi), DRC and Tanzania / Rwanda / Burundi / Uganda (rift valley lakes), and so on.

It was also quite common for initial control to be established over an existing kingdom as a European protectorate (e.g. the Kingdom of Buganda in Uganda, in Botswana, or in Malawi). These would later be extended when they came under full colonial control to the aforementioned geographical boundaries.

As for why Mali and Niger aren't one country, the territory of Mali was colonised by France around 1880, and Niger from 1900 - 1922. They were both made part of the French Federation of West Africa, but remained separate entities within this. Generally colonial administrators were reluctant to put too large of an area into one single entity, for whatever reason, though they were sometimes put into federations as in this case.

tl;dr The borders are a result of competition and compromise between European countries for African territory. Initial claims were generally based on older coastal trading posts, on existing protectorates over precolonial kingdoms, or on agreements e.g. at the 1884-5 Berlin conference. These claims were then extended to natural barriers like rivers and mountains if they were available, or geographical coordinate lines if they were not, in a process of settlements between competing interests of and between European countries or for administrative convenience.