Spain and Portugal repeatedly occupied different cities and coastal areas since the reconquest before being driven off. I don't think Spanish every became a common language but some locals would have definitely picked up the languge. Also during the recongquesta, a lot of the people who fled Spain to Morocco would speak Spanish, so perhaps you had neighberhoods and villages where Spanish was the common language.