Do you have a specific point in medieval history in mind? Armor changed over the period (remember, it spanned a thousand years), and varied from place to place as well. I'm going to assume you mean Europe?

Anyway, a good starting point for this is the Bayeux Tapestry. It is important to keep in mind that the tapestry is a piece of propaganda^1, so there's a chance that there are some scenes where the subjects are in full battle armor, but may not have been in actuality (and vice versa), to lend "cool factor" to the scene.

Here you can see Harold Godwinson and his entourage heading home from a meeting with King Edward. They don't appear to be wearing any armor, despite Harold being an important political and military figure at the time.

And here is William I at a feast after landing in England. As you can see he, too, appears to be out of armor.

So, from the tapestry, we can see that people, at least those in England and Western France, didn't do all their travelling in armor.

I'm not sure what you mean by "without a direct military purpose". People used armor to fight. For pilgrims, illustrations from the Canterbury Tales didn't feature much in the way of armor. Here is a depiction of Chaucer as a pilgrim from the Ellesmere Manuscript and you can see that he doesn't look particularly reday to throw down.

Though, we can look at the Crusades which were, according to the Crusaders, an armed pilgrimage and see a lot of depictions of people traveling with armor on. Here is a great textbook about the Seventh and Eighth Crusades that has a lot of examples. Crusaders are almost always depicted decked out for battle. Again, though, it's unclear to me whether or not this is a completely factual depiction or if it was artistic license. Perhaps somebody who took a class with this book rather than just reviewing it alone, or actually does professional research of the Crusades, can answer that in more detail. And this is, obviously, a war so it doesn't really come close to being sans military purpose.

EDIT: It occurs to me that the direct military purpose is referring to the people, not the armor. In that case, people traveling with a host that didn't have a direct combat purpose (most would have a "military" purpose, at least from a logistic perspective, the exception being people like unarmed pilgrims traveling with crusaders or poets/clergymen/etc.) generally wouldn't have armor to need to change in or out of. During the First Crusade, the host was accompanied by a large number of unarmed pilgrims, and the host itself was comprised in large part by volunteers, so what differentiated a person who did or didn't serve a military purpose was, basically, whether or not they showed up armed. Furthermore, armor was prohibitively expensive throughout the Middle Ages, so even people who did serve a military function often didn't have armor when they set out.

1. Sort of. I'm not fond of that word choice. The tapestry wasn't really public enough to be called a piece of "propaganda". I just mean to say that it has some biases that lead to some probable inaccuracies. It attributes some acts to its commissioner (Odo of Bayeux) that other sources don't attribute to him. So, anything in it has to be taken with that sort of thing in mind.

I remember similar questions being asked before -- unfortunately, a good part have top comments deleted by users from what I can tell (not mods). Two threads might still be of use, though:

• In shows like Game of Thrones, characters wear their armor almost everywhere. How accurate is this portrayal?
• How did knights poop?

Well, pilgrims and other kinds of non-military travellers would most likely not wear any armour at all, and military personel would only wear it if they thought the situation demanded it, for example, if they were relatively close to the enemy, or maybe, maybe, if they were travelling through an area notorious for banditry, like the Parisian Basin after the Jacquerie, when the famous brigands and routiers made camp there. Otherwise, it would only be worn in battle.

Some people are asking about when plate armour developed: it was a gradual process that started roughly in the 13th century with the incorporation of small pieces of plate onto the chainmail, and eventually incorporated cuirasses in the 14th century, until it covered almost the whole body by 1420 and onwards, if not a tiny bit earlier.

I have a question along the same lines:

Would Roman Legionaries typically wear their armor the majority of the time? I know this is a very broad question but any insight would be very interesting!

Follow up questions which I'll limit to just middle ages - when it comes to leather armor, would that have been more 'wearable' than plate or chain, or serve as an underlayer that could provide some protection when full on battle armor wasn't needed? Or would it also have been very heavy and cumbersome? To what degree would people in the middle ages have worn armor when they were off duty or perhaps making a regular journey - would it be normal to have something like a leather jerkin to avoid casual stabbings and the like?

One of the items in The Portable Medieval Reader by Ross and McLaughlin is an excerpt from a First Crusader document that has always stayed with me for its vivid hit in the head on just this subject.

Here the Crusader army is, riding along in Anatolia, when they meet an army of Turks. The light outriders of the Turks begin to shower them with arrows while their light cavalry goes out to chase them off. In the meantime, the rest of the army stops, hastily throws up the tents, and the knights change into their armour. The Turks are doing about the same thing, and whoever gets armoured up first gets first hit in on the other side, right? Mind you, this is armour pretty much still out of the Bayeux, hauberks to the knee, helms, shields, and a weapon or two.

To me, it was permanently engraved that neither side was ready for battle while on the road. And these were invader and defending armies!

Ambushes are also a good case. No, unless they suspected attack, travelling parties did not wear armour. So if you could pull off your movements in real secrecy from the local peasants, your armoured party of attackers could almost guarantee success because the guys you bushwack aren't going to be in anything but normal clothing while you're much harder to injure. In 1168, William Marshall was wounded and captured when riding in his uncle's train and they were ambushed. Ambush is a great advantage if you can pull it off.