Part of the answer resides in your question: "especially if you weren't in any sort of position of wealth and power." Well, for sub-Saharan Africa (and even chunks of the Maghreb), the answer was often that you didn't go on hajj. In addition, until after the 15th century, Islam was not the religion of the masses in the western Sudan. It was an urban, elite, learned creed, often highly syncretic, and adaptable. Those who went on hajj were rulers, scholars, wealthy merchants (who had additional motivation!), and others with the means to pay for the long passage via caravan from Gao or Timbuktu via Tadmekka towards Egypt. The result was that a lot of professing Muslims didn't go; having the prefix al-Hajj was a notable rarity. [e: The actual poor masses didn't go, of course, because they tended not to be Muslims at all.]

That said, these areas weren't truly isolated, because people could actually go via Berber trade caravans and reliably so. Those who did go often transferred new ideas, made new personal links, and generally strengthened bonds within the Islamic world. If you want to see the kinds of movements people could undertake, and who might undertake them, Ibn Battuta's Rihla is really one of the key texts of the era; Battuta's notes and recollections, written down from dictation, are quite detailed and can tell you a lot about the nature of travel in the Islamic world. It was remarkably safe on the whole--pirates notwithstanding--but it was not cost-free. Someone like Battuta, an Islamic scholar, could count on (or complain for!) costly hospitality at his destinations, which aided the movement of those kinds of well connected people.

One further note is perhaps in order: the monsoon system of eastern Africa made the pilgrimage process for highly-placed Swahili coast families easier than for many who might be closer as the crow flies. With usually reliable seasonal winds, and steadily improving ship technology, piracy was the only concern--and even that was not exactly a big worry for pilgrims until the Portuguese popped up (at which point it became terrifying). Sending children to study in Arabia, or elsewhere in the region, was possible despite the seeming uncertainties of sailing most of a continent's coast. The East African coast was quite vibrant by the early 15th century, and very well connected, but again the majority of Muslims in those societies were not poor farmers--the rise of mass Islam towards the East African interior (south of Somalia) would be a much later and patchy development compared to the rise of mass Islam in West African sudanic regions.

I'll throw out Randal Pouwels and Nehemiah Levtzion's edited The History of Islam in Africa, although they only talk about difficulties and connections via the hajj at various moments in the early chapters, but they do get into the very partial (and class / urban) nature of Islamic conversion initially in Africa south of the Sahara.