I am familiar with the limited Portuguese-French conflict in the 16th century over Brazil with battles of Rio de Janeiro happening, but as I see there are also 1700s raids on Rio de Janeiro, and I am not sure if OP is asking about this. My answer is limited to 16th century. I am sure while some of the stuff would still apply for the later phase, most of what I write is specific for this time period.

Why couldn't mainland Portugal be attacked? in the 16th century

For the most part of 16th century (and maybe even later) the colonies were of secondary importance to European powers. Not that they did not care, far from it, but most of the powers were at the same time pursuing their own goals firmly rooted in classical European politics. French and Habsburgs (who ruled HRE and Spain) in particular had spent most of first half of 16th century engaged in endless back and forth over who held swayed in Italy, dubbed the Italian Wars. Habsburgs also had to deal with the rising of Ottomans and their assaults in Mediterranean and Balkan/Central Europe. The second half of 16th century brought fallout from the Reformation. France was engaged in civil war of their French Wars of Religion, and Spain became bogged down with Dutch War of Independence, and the Ottoman attacks were at their peak. Both Spanish and French had majority, if not all their war efforts engaged in those conflicts, and they had little resources to spare for alternate pursuits.

Portuguese were heavily involved in their pursuit of overseas empire in Morocco, Africa, Brazil and Indian Ocean and Asia, whereas Asia and Morocco were their prime target and consumed most of their efforts, bringing them to the verge of over-extension. However, very importantly Portugal was in a very comfortable geo-strategical position with them sharing land border with only Spain. Spain and Portugal had a rivalry and disagreements over borders in their colonial holdings, however both were preoccupied with other conflicts and had no desire to fight each other and actually pursued policy of cooperation and marriage alliances. King Manuel of Portugal was married to daughter(s) of Ferdinand and Isabella, while his son King Joao III and Spanish King, HRE Emperor Charles V solidified the connection by double marriage with marrying each others sisters.

This meant that the only country that could attack Portugal by land was in effect in good relations with them and certainly wouldn't allow their major rival France to attack Portugal through their territory. In fact, it is more probable they would aid Portugal in case of any attack by the French (in a manner they sit fit of course).

This leaves the only option for the French could possibly use is an invasion by sea. But honestly, logistics of operating and transporting armies in the 16th century was not on the level for this to be a realistic option. Few states in those times actively built a serious navy, and Portugal was one of them, while French less so. At least not enough to actually preform an offensive against the Portuguese.

Navies, ships, sailor wages were incredible expensive, and it was incredibly costly to upkeep. Even if France succeeded to neutralize the Portuguese navy, they would still have to tansport a huge mass of troops by sea, a very risky and dangerous prospect, especially given that Spanish ports would be closed to them. And having huge mass of troops confined to tiny spaces of ships for a longer time is a recipe for catastrophe. Disease, malnutrition, scurvy could decimate an army. A single storm could wipe it out.

Example of attempted invasions in the end of 16th century were the Spanish Armada and the following English counter armada of the 1588/89, both of which were complete failures.

Again, even if French managed to land forces in Portugal, the outcome of this wouldn't be straightforward as you would think. Generally speaking, while larger states had an advantage, they wasn't really a simple way to transfer their advantage in size and manpower into a larger and better army. And then to operate this army successfully far away from home. Habsburgs struggled greatly - and ultimately failed - to contain the Dutch Rebellion, and they actually had a land connection to send troops and resources there. Hypothetical French forces landed in Portugal would have only the sea route to get supplies, reinforcements, money for wages to keep operating, and they would struggle to do so.

TL;DR Portuguese were protected by Spain from land attacks, and naval invasions weren't really an option in 16th century