North Africa had long been a part of the Roman Empire, and there were even Roman emperors who came from North Africa. It had also been a Christian land before the mohammedan invasion. So why did the Spanish stop after taking back Granada?
What if America had not been discovered? Would the Spanish have redirected their resources towards taking back the Mediterranean Basin? Would they have been successful?
Well the assumption in your question isn't entirely correct: the Portuguese and Spanish did take back part of North Africa after driving the Muslims out of Spain.
The Portuguese took Ceuta as early as 1415, almost 80 years before the Reconquista was completed. It went over to Spain in 1668's Treaty of Lisbon and has remained Spanish ever since. Melilla was taken in 1497. Tangiers was conquered by the Portuguese in 1471.
What these cities had in common is that they were port cities, which were easier supplied and defended not unlike the other African trading posts that comprised the Portuguese empire. They were also of very high strategic worth as they gave control over the Straits of Gibraltar.
A major reason why the Portuguese and Spanish could not further expand into Northern Africa, apart from lack of necessity and non-religious motives, is the emergence of the Ottomans in this period. Not long after the Reconquista the Ottomans managed to secure control over the entire Eastern Mediterranean and became a huge naval power. Their dominance would not be broken until the Battle of Lepanto in 1571. Even after this they remained a potent force and adversary to a Spanish Empire that was more focussed on the Americas because they were more profitable.
I'm not going to delve into 'what ifs' since I question the historical value of such questions. Ottomans dominance over the old routes towards Asia was a major reason for the Europeans to venture West in search of a different route though.
/u/socrates55, see this post.
The monarchs, nobles, and enterprisers of Spain's composite monarchy in the early modern era did have designs to continue the Reconquista to north Africa. There were various plans and attempts to realize this, with widely varying degrees of success. And just like with other endeavors, there were ebbs and flows.
In the 1500s, Spain was able to conquer several key positions and maintain a string of garrisons in North Africa, the idea being to disrupt Ottoman galley operation and enable that of the galley fleets of their allies. Galley warfare was the dominant marine and coastal way of war, and it was very resource extensive as one has to maintain and replenish crewmen. As such, maintenance of friendly outposts and denial of enemy access to their own outposts was vital.
The high watermark of Spanish expansion into North Africa was in 1511, when they controlled string of outposts (presidios) from Melilla, Mers el-Kebir, Oran, Budia, and going as far east as Tripoli. The Spanish were able to force local rulers into vassal status. Of course, Portugal also had expanded into North Africa, taking Ceuta and Tangiers among other outposts. For example, here's a set of stunning photos of the presidio on Peñón de Alhucemas in Morocco. And here is a photo of the Melila presidio of which some walls and redoubts survive today even as a city grew around it.
Unfortunately for Habsburg Spain, the Ottoman empire and her agents were able to push back, leading to the loss of most of these outposts. This is why the Ottomans were hell bent on taking Malta, even if the Knights of Malta heroically defended their island outpost in 1565. All this spectacularly coming to a head in the Battle of Lepanto in 1571.
In between, there were major attempts to re-take lost outposts, one of the most important was Charles V's attempt to take Algiers in 1534. He assembled a large galley fleet, but was beset by poor weather that led to the loss of a significant portion of his force. Notably, a disgraced Hernan Cortés -- having been denied the vice-royalty he sought in the Americas -- was part of this expedition, even if he was not invited to the war council. His sense of honor slighted, Cortés asked to be allowed to make a last stand to the death, but this too was denied.
But why did Spain fail to achieve her goals in north Africa? As I wrote here, a lot of Spain's plans were thwarted by heretics in Germany, in the Low Countries, and in England. Over time, low level warfare and raids continued in the Mediterranean coasts, both sides raiding for loot and captives. The continuing enmity and lack of security in the western Mediterranean eventually led to further and further pressure on Moriscos in Spain, culminating in the expulsion of Moriscos in 1609.
The history of Spain's north Africa is truly overlooked but has been asked about twice here recently. I wonder what triggered the interest.
References? This is a bit complicated. You can start with the highly readable Pirates of the Barbary Coast by Tinniswood which covers the conflict at large but little on the presidios themselves. There is a recent collection of academic essays Spanning the Strait which covers many aspects of what I wrote above, and is probably the best starting point.
