Is it the reason most of the countries are in extreme poverty?
This is a complicated question, Africa is a very large continent and each country has its own issues that it has to deal with. I will refer to some of the processes that took place in Central Africa (Congo, Zambia, Angola and Tanzania area). Many of the issues here may not apply to other areas. There might be quite a few state names here that very foreign. Please ask if you want clarification on particular state.
Pre-Colonialism
Currently we are very confident that there were many states in central Africa before colonialism. For the sake of simplicity I will divide them into three catagories: European (mostly portugese) west coast settlements such as those at Luanda and Benguela. Further inside the continent Central African polities like the Luba and Lunda empires, the Kuba kingdom and the Bemba state. On the opposite coast we have the Arab Swahili coastal settlements (of Omani origin).
Both coastal settlements are involved heavily in the transportation of slaves in central Africa. The Arab Swahili are trading in the Arabic world and the Portuguese send most of their slaves to Brazil. The driving force here is commerce, as demand increases on both sides each try to get further into the interior to get more slaves. The African states, however, rely on royal monopolies to ensure the stability of the state. In this sense African states are actively trying to prevent Europeans from reaching the interior, we see this in the accounts of Portuguese merchants trying pass through Kasanje. The leader of the town (the Kinguri) threatened death on any who tried to pass through. The kings of trading states would take the fruits of commerce and distribute them among their subordinates ensuring loyalty.
This puts the three sides at a stand off, the Arab-Swahili and Portuguese want royal monopolies broken to drive down slave prices, the African Kings want them kept strong to get better prices for slaves. In the mid 1800's most European countries ban the slave trade, this causes the slave to be replaced with ivory in most European transactions. Once again the stand off between royal monopolies takes place, very similarly to before.
Not all states are involved in slave trading. The Lozi and the Kuba peoples were actually slave importers a fact that is very important in the onset of colonialism. So in the mid 1800's the Europeans and Arab-Swahili become much bolder in thier advances. This is the Era of David Livingstone and Henry Moreton Stanley. The Europeans are moving through and breaking up royal monopolies. This causes the collapse of many states such as the western Mwata Yamvo Lunda collapse after the royal monopoly is bypassed and Ivory becomes a free for all market.
In the East Coast the Arab Swahili too are making their way into the interior. The head of this charge is a man named Tippu Tip, a notorious slave trader and pillager. Tippu Tip and the Arab Swahili tended to back local chiefs (giving them guns) and encourage them to fight, taking the bulk of spoils for themselves. Tippu Tip plundered an immense amount of Ivory and retired a very rich man. A lot of states collapse under this pincer movement of commerce, the states that survived generally imported slaves (meaning royal monopolies on slaves were irrelevant) or had unusual state formation patterns (a type of federation was used by the Bemba)
Perhaps inevitably we reach a point where the Arab Swahili and Europeans meet each other around the center around the Katanga region. A lot of the Arab Swahili retreat back to the coast with the immense riches they gain, some stay and are incorporated into colonial states eventually. What both the Arab-Swahili and the Europeans leave behind is a lot of guns and a legacy of violence, there is massive displacement of peoples across the region due to being forcefully removed from their homelands.
Colonialism
Now with this in mind, massive displacement of peoples, a lot of weapons in the area, grudges between states and villages the Europeans come in force and start creating colonies. Peoples who actively hate one and other forced to live under the same state, the Lunda state is current halfway between Tanzania and the Congo. Royal monopolies are completely abolished in favor of colonial commerce. Traditional power structures that survived such as the Kuba, Lozi and Bemba are superseded.
When the mobilisation starts for decolonization a lot of the groups are ethnocentric. For example in Angola, the MPLA were a largely Ambundu organistion while UNITA was a largely Ovimbundu group and the FNLA were mostly Congolese. They also organised on ideological grounds, but the ethnocentrism is still there.
In answer to your question, the Slave Trade and Colonialism did cause a lot of problems in Central Africa. However, it was the violent introduction of commerce into the area by Europeans and Arab-Swahili that initiated the decline of many states in the region. Colonialism too had a decisive impact on the many of the areas involved by the processes that caused many of the problems were already in full swing before the onset of colonialism.