On the one hand, yes, it would be fairly unlikely for you to be recognized if you went to town a reasonable distance away. However, as in the case of Martin Guerre's imposter, Arnaud du Tils (See Natalie Davis, The Return of Martin Guerre, which I will draw much of this statement from), even going a fair distance and adopting a false identity didn't leave you unidentifiable forever.

To truly answer this question, however, we have to do a little bit of probing into the lifestyle and worldview of the common person in medieval (and early modern) Europe. Medieval people did not view "immigration" in a positive light, in the slightest; most believed that the amount of food and resources available to be finite (fairly true prior to the Agricultural Revolution), and that the addition of new people to a village or town would be a drain on the resources due to them and their children. Most people worked the same job as their parents, and their parents' parents: towns only had so many fields to farm, so many metal goods to smith, so much ale to brew, etc. With few people having access to the money economy, your entire economic wellbeing was tied to the place you were born, and your social position among them was understood and fixed; if you left, you could bring nothing with you that would sustain you for long, and would find no place to call home. Few people ever traveled more than a few miles from their place of birth, and even fewer would ever seek to immigrate. Most people who moved around were in terrible shape, often relying on charity and the rare oddjob to survive. You would be hustled from city to city, and town to town, with authorities and peasant folk alike eager to move you along.

This is why exile/banishment/outlawing were grave punishments in the medieval world, but no longer carry the same sting. Exile, for the medieval person, was to be completely uprooted from your identity, to be cast from any semblance of security into a life purely at the mercy of others. They "sta outlawed" because, once outlawed from their home, there was no place else they could call home.

Further reading:

Natalie Davis, The Return of Martin Guerre

Robert Jütte, Poverty and Deviance in Early Modern Europe

Richard M. Wunderli, Peasant Fires: The Drummer of Niklashausen

Disclaimer: I don't mean to say that no one migrated, because plenty of people did and some people were successful or, at least, established in their new homes. However, it was generally considered an act of last resort, when your life was already in grave jeopardy (whether from starvation, pillaging, legal punishment, etc.)