Why didn't Spain see Portugal as a threat and invade them?

Portugal and various Spanish kingdoms (Leon, Castille, Navarra, Aragon, and their various union states) had waged a whole bunch of wars between each other during medieval and early modern years over various reasons, often involving attempts to take over the other's country. What might surprise you was that there were several real attempts by Portugal to take over the Castillian throne too!

Medival concept of conquest

But first you should really be aware that in the medieval and early modern European world you didn't really just invade cause you think someone was a threat and especially you couldn't just annex countries. You needed a reason to attack, you also needed a strong, legal, claim to the throne to claim it, and most importantly you needed a strong army to actually pull it off. Of course, that didn't really stop anyone attempting, even when the legality was a bit iffy. Portugal's proclamation of independence in 1129 and kingdom in 1139 was one such case. But successful military actions brought a de facto independence in 1140s, and long lobbying at the Papacy( through crusading actions as well as showering the local bishops with lands and rights) brought them recognition and political, de jure , independence in 1179.

This is a gross simplification but explaining overall Iberian medieval politics is too complex for me to try to unravel, but basically in middle ages, Iberian kingdoms usually respected each others independence status (borders not so much) until they had a strong claim to the other throne. This wasn't uncommon as royalty of the various kingdoms often intermarried and had a lot of illegitimate, but recognized and respected children adding to the mix.

Crisis of the 1383-1385

In 1383 Portugal was in one such situation when king Ferdinand of Portugal died with no sons and his daughter and heir was married to king of Castille John I. What was worse king Ferdinand for a long time led a serious of wars against Castille, trying to put himself or one of his allies on the throne. The wars were not only unsuccessful, but almost disastrous economically to the Portuguese. Especially the last one which was a complete defeat of the Portuguese. But the Castillians couldn't just take over Portugal despite the win, but arranged a marriage which would basically put the Castillian on the throne in the next generation.
So when the king Ferdinand I died, his daughter and her Castillian husband were suppose to inherit the crown. The Portuguese populace was unhappy (nobility was divided, but slightly more pro-Castillian) and proclaimed their own João I as new king of Portugal. João was illegitimate half-brother of Ferdinand, and master of knightly order of Aviz. He was actually the half-brother of Ferdinand with the least claim to the throne but the other brothers were both in Castille (and then captured) so not available and not viewed favorably anyway.

Castillian king John I then sent armies to Portugal, as he was legally the one in the right (even though Portuguese lawyers rejected his legal right). His armies were surprisingly soundly defeated in few battles, most notably Battle of Aljubarrota in 1385, which (similar to battles of Crecy or Poitiers between English and French) featured much smaller Portuguese force using dismounted man-at-arms and archers in a prepared defensive position to destroy much larger enemy force. This victory sealed Portuguese independence and brought new dynasty to the throne: the house of Aviz which has strongly and securely ruled Portugal through the Age of Exploration.

House of Aviz and war of Castillian succession

During this period (15th and 16th century) house of Aviz was secure in Portugal helped with the wealth from Africa trade, and even in 1470s made a claim on Castilian throne. King Afonso V married his niece, who was one of the heirs of Castille, and opposed (future queen) Isabel and her husband Ferdinand II of Aragon. Battle of Toro decided the winner of the crown (Isabel). At the same time Castillians made an effort to take over the Portuguese African trade but were defeated on sea, which in turn sealed the control of Africa trade to Portuguese. In the following years, the kingdoms of Portugal, and now Spain (union of Castille and Aragon), were in rivalry, but not really "threatening" each other. The Treaty of Tordesillas separated the zones of influence of each nation which provided both with plenty of opportunities and area without actually coming into conflict. One of the potential hotspots was Moluccas (Spice islands), which could fall into both Portuguese and Spanish areas, but the nations resolved this issue with diplomacy (and payments)

Spanish throne soon went to Habsburg dynasty which brought the Spanish involved in their various European conflicts most notably Netherlands and the rising Protestant issues. Still in 1520s, Portuguese king João III felt either envious or threatened by Emperor Charles V, and went to great deal of effort to organized a double marriage, where he married Charles V sister and Charles V married his sister, arranging peace for Portugal ( btw In order to do so João paid a massive dowry to Charles, which might have severely weakened the Portuguese state according to some historians sending them on the downward spiral)

Finally the spanish invasion

After João's III death the Portuguese throne went to his grandson Sebastian. In 1578 Sebastian, young and foolish, personally led an expedition on Morocco where his army was defeated and he disappeared. Having no children the Portuguese throne went to Sebastian's grand-uncle Henry, who was also old and happened to be a cardinal in the church! The pope did not allow him to be released of his vows of celibacy (because of politics) so when he soon died the Portuguese throne was open.

One of the claimants was Phillip II of Spain, as his mother was sister of João III. This was a good enough claim, not the best one, but more then enough, but he had some opposition in Portugal (which wasn't really unified) Spanish army destroyed the army of the rival claimant and was pronounced king of Portugal in 1580, under the conditions that the Portuguese kingdom remains independent from Spain, but under the same ruler, which was more or less honored under Philip II. So as you see, the Spanish DID in fact "invade" and take Portugal

This new joint state: the Iberian Union controlled much of the New World as well as a lot of Europe, but suffered from consequences of such size and power. Various wars, like the Dutch War of Independence (80-years war) and then Thirty years war resulted in the Dutch taking over some of Portuguese outposts in Africa, Brazil and East Asia.

Restoration of Portugal
New king Philip III had financial problems because of these wars and raised the taxes and tried to incorporate Portugal under his central rule. This resulted with the Portuguese to declare independence in their Restoration War. Serious of Portuguese victories, general Spanish exhaustion from multiple wars and simultaneous Catalan Revolt led to Portuguese success and recognized independence by Spain in 1668. In the following years, both kingdoms Portugal and Spain where nowhere near the forces they used to be and remained generally respective to each other's status.

Summary

Spain in fact did try on several occasions to take over Portugal, as well as Portugal on some occasions trying to take over some of Spanish kingdoms. In other times they were friendly neighbors or just rivals who fight but don't try to conquer each other.
Spain succeeded taking Portugal in 1580 but the multiple wars Spanish were having brought them to ruin and allowed Portugal to reclaim independence in 1640s.