It seems like after the reconquista ended they were both leading the world in military and naval power for a couple centuries. Why didn't they cross over to Africa and continue gaining land for their kingdoms much as the ottomans had done in Eastern Europe?
They did try and conquer North Africa, even before the Emirate of Granada was destroyed. In 1415, the Portuguese sacked the Muslim city of Ceuta in Morocco and established a garrison fortress on the coast. In 1437, the Portuguese suffer a bad defeat, trying to conquer Tangier. Prince Ferdinand of Portugal was captured during the humiliating evacuation of the army that was besieging Tangier.
After Granada fell, the Spanish did expand into North Africa. The high water mark was between 1508 and 1512, when Spain controlled Algiers, Bejaia, Tripoli and Djerba. However, the resurgent Ottomans struck back and in 1512 the Barbarosa brothers, Uruj and Khizr, began to roll back the Spanish North African empire. Algiers was finally retaken in 1530, by Khizr, the younger of the Barbarossa brothers.
The last major Iberian attempt to conquer North Africa was led by King Sebastian of Portugal, in 1578. It ended very badly, with Sebastian and most of Portugal's nobility being killed at the battle of Ksar-el-Kebir.
Between 1415 and 1578, both Spain and Portugal fought many wars in North Africa, but eventually the Muslims were able to prevail. The full details can be found in "The Last Crusaders: the hundred year battle for the center of the world" by Barnaby Rogerson.