I can only answer in the case of Denmark in high medieval times, and only in the case of written law. I hope it will be satisfactory.

First of all I will give a short introduction on danish law at this period in time (1200-1350-ish). Denmark was divided into 3 different areas where different laws became official: The law of Scania, the law of Zealand and the law of Jutland - these laws were different from each other, but their differences should not be overstated I think. They mostly operated in the same way. But take notice that the laws I'm going to discuss often were a little different from law to law. Add to this that there also was a seperate religious/canonical law enforced by the catholic church which mostly dealt with sexual crimes or other religious crimes.

So the different laws that came into use in Denmark during the high medieval period (1200-1350-ish) used what can be described as mostly three different ways of punishing criminals. The first is corporal punishments such as mutilation or executions - these kinds of punishments were usually only given out to thieves (thieving was considered heavily taboo at this time in Denmark). The second would be fines - these were the most popular kinds of punishments in the danish laws. The last would have been an annulment of the criminal's rights such as condemning the criminal to become fredsløs. This meant the criminal lost all rights according to the law, their properties were seized and they could be killed with no punishment. (I think this is very similar to exile in the anglophone world).

As you can see, imprisonment, and more relevant to your question, forced labour which is what i would classify the punishment given to the criminals manning the Wall in Game of Thrones didn't really exist in the law texts of medieval Denmark. Why not you may ask? Put simply, the danish state was very much underdeveloped compared to our modern notions of a state. There simply wasn't a functioning state apparatus that could keep an eye on prisoners or forced labourers at the time.

However this is just from what the law texts tell us. We know very little about how the law was actually dispensed by the herredfogeds (think: the judge/sheriff of a juridiction). From texts from the 1500s we know many criminals weren't able to pay fines right at the moment they got them, but had to pay them off over time. This would be similar to forced labour, but they wouldn't be shipped away from society like they do in Game of Thrones.

Now if you think of the Wall as a place to just remove people from society, the most similar punishment would have been fredløshed/exile. There was no state apparatus to move these people away from society or to keep them there, but the consequences of fredløshed would usually mean the criminal had the flee - either to another part of Denmark or to one of the neighbouring kingdoms.

Canonical law also had a way of isolating people from the society they lived in through different types of excommunication. I'm not a specialist on this subject, but to my understanding, this was usually done through a refusal to give sacraments to the excommunicated person. They would be spiritually isolated from their society, but not physically like with the Wall.

All in all, no, there was no institution like the Wall in Game of Thrones. The state system simply wasn't developed enough in Denmark, and even if it was I don't think there necessarily would have been a similar institution.

Sources:

"En dansk retshistorie fra middelalder til grundlov" by Kjær and Vogt

"Kirkelig disciplin og social kontrol" by Ingesman

I don't watch GoT, so I can't speak for the 'wall', but banishment was a regular form of punishment in early modern Europe. What's important to understand is that most cities in this period are surrounded by an enormous wall with multiple gates, that allow the society to distinguish between 'citizens' (those within the walls) and 'strangers' (those without). That's why most of the big execution sites are outside the walls - there is dishonour in not being part of 'civil' (ie of the ciy) society. There is plenty of movement through the gates, but the ones who were welcome were mostly merchants/farmers coming to sell their goods at market - many others were instantly viewed as suspicious. To not belong to a city (which is what happens when one is banished) generally meant that the person belonged nowhere: could not earn a living, had no home, was usually reduced to beggary. So banishment is a serious punishment; it's why Romeo freaks out when he hears he is to be banished (it's not just about missing Juliet). There were also ceremonies, usually during the visit of a high-ranking noble or monarch, where they could show their mercy by allowing formerly banished citizens to return: they'd wait outside the gates and beg to be allowed in.