First off, the idea of neutrality is somewhat problematic from a medieval perspective and, usually, it either wasn’t practical or recognised. A medieval ruler usually reacted in three manners if it had a foreign army marching through his/her lands: suffer it, oppose it or support it.

So, the passing army would usually rely on what they found in the land they found themselves. If the army was an overwhelming force they might force the local ruler to provide supplies or demand homage of some kind under duress. In such a case, the local ‘neutral’ ruler would suffer the foreign army as long as it existed as a threat.

If the local ruler wasn’t directly involved in the conflict but was sympathetic to the enemies of the passing armies, they might attempt to deny access or harass or stress their supply routes. Thus passively opposing the passing army.

In other cases, especially in Italy, merchant republics may allow passage to the kingdom of Sicily to invading Angevin armies in exchange for support or money. The success of an expedition was heavily dependent on ensuring the loyalty and positive diplomacy with local regional medieval polities. On some occasions, in Italian towns, Holy Roman Emperors and French kings when they passed through Italian lands with their armies for some reason or another, they were asked to resolve local legal and political disputes.

I can cite an example whereby the city of L’Aquila had denied and allowed access to armies invading the kingdom of Sicily through the strategic route through the Abruzzi. The city was of Angevin faith and became an autonomous polity hegemonic in the region. Consequently when Charles d’Anjou required its help it rose up to support his claim to the kingdom of Sicily. Similarly when Conradin descended into the south of Italy and attempted to pass through the Abruzzi L’Aquila denied him access and supported Charles d’Anjou in the battle of Tagliacozzo. The importance of L’Aquila as the guardian of the strategic passage in the Abruzzi reached its apex in the 14th and 15th centuries.

So, to answer the question, it depends if prior agreements had been made and where the local ruler’s sympathies lie. If there were none, then the act would have been seen as an aggression.

Hope this answers the question!