Why is it the North Africa is usually theorized as part of the Arab world/Middle East rather than the actual African continent???

In addition to the answer that /u/Dinocrocodile linked to, I'll also link to my answer in this thread.

I think that Hegel is important to bring up here for 2 reasons. Firstly, his Philosophy of History reflects a certain time and place with European's engagement with Africa. The book comes out of university lectures from 1828, '29 and 1830, very late in his life. At this point, Western European societies like the Portuguese, Dutch, English and French had been trading with coastal African societies for about 4 centuries, and the Atlantic slave trade had grown and was on its way out (Britain outlawed the trade of slaves to her colonies in 1807, and would outlaw slavery in those colonies in 1833).

So, there were already areas of coastal European control, like Elmina castle, St Louis-Goree or the Cape Colony. But on the other hand Europeans were just beginning to travel into the West African interior, and explorer missionaries like Livingstone would not reach Central Africa until the 1850s.

So, Hegel's characterization that "True Africa is a continent without history" was based on a very limited understanding of the continent.

This brings us to the second point, that his ideas of unchanging and timeless African societies, and his division of the continent into "European Africa" (North Africa), Egypt, and "True Africa" (sub-saharan Africa), were articulated and became influential in time to shape the education and worldview of upper-class Europeans that would lead their countries in the period of the Scramble for Africa roughly from the 1870s-1900.

Hegel's ideas would also form part of the intellectual backdrop (along with Auguste Comte, Herbert Spencer, and Emile Durkheim) for generations of ethnographers and anthropologists in the colonial period who would observe and document African societies as service to the effort to make colonial administration efficient and "rational".

With Hegel's idea that North Africa was more "European" and therefore more modern baked into these anthropologists and administrators' worldviews, French and British colonial policies thus ranked Arab (North African) populations higher on the "ladder of civilization" than Black Africans, and Muslims higher than pagans*. Thus, different administrative, missionary, and economic prescriptions were proposed for these different populations.

So, this perception of difference was reinforced in European colonial governance and expectations during the colonial era.

Why has this dichotomy remained so persistent in the post-colonial era?

I would chock this up to intellectual trends in the American academy in the 1950s, with the rise of the Area Studies paradigm. Area studies is closely tied to the early Cold War, and a perception in the United States that universities needed to produce the diplomats and experts to allow the US to engage with the Third World, and win the ideological contest with the Soviet Union and China being waged in the recently decolonized nations of Africa and Asia, as well as Latin America**.

To encourage familiarity with the thought processes and culture of an area of study, language study is often made part of an area studies program's portfolio. For instance, African Studies programs will often offer Swahili, Yoruba, Amhara or other language classes. Latin America Studies may offer Spanish or Portuguese classes.

As you are likely aware, the predominant language in North Africa is Arabic***, and so linguistic concerns had a role in encouraging many schools to lump North Africa into Mid-East Studies programs, probably also influenced by the received wisdom of Hegel and Colonial era history.

To state the obvious, having an ability to read source documents in Arabic is tremendously important to the historian of post-classical Northern Africa, and so they likely engage with the Middle East Studies program to take courses in Arabic. Thus, the way that scholars view connections or discontinuities between North Africa and Sub-saharan Africa, or North Africa and the Mid-East can be put through a filter, which they may or may not question.

*of course, "pagan" as a religious category itself reflects Europeans understanding of these belief systems as similar to ancient Greek or Roman pantheons. There are many ways that such assumptions are problematic.

**I should note that the area studies paradigm interacted with the disciplines of history and anthropology, but also other fields like literature, political science, economics, and others.

*** Of course, there are prominent Berber speaking populations in Algeria, Morocco, as well as other linguistic minorities in the region. Also, Arabic is spoken (often as a 2nd language) by populations in West Africa and in the Horn of Africa, while Arabic is usually a 1st language in North Africa.