Disclaimer, I don’t know a ton about marches outside of Medieval Britain, so hopefully someone can fill us in around German, Spanish, French marches etc. But I’ll do my best with what I know!

The Welsh Marches between core English areas and the domains of the welsh princes were originally set up during the Norman Conquest, as Norman nobles/adventurers swiftly built castles and expanded into areas previously demarcated as under welsh control. Granted, the Normans set up castles practically everywhere they could, densely populating England with their Motte and Bailey style forts - simple to build and helpful in establishing control over a rebellious countryside.

In the welsh regions, the Normans originally expanded quite rapidly, but a certain equilibrium established itself where the two groups would raid and battle, allying and betraying each other in dizzying fashion.

In the safer, ‘core’ English lands, William originally granted out lands/honours in disparate patches that were difficult to rule or manage from one place - its often cited as a strategy to keep his barons under control, as they wouldn’t hold compact, large blocks of land from which they could develop personal power bases and be a significant danger to the stability of the kingdom. What is more, barons in say Kent were (after some time) not allowed to build their own castles or privately war on their neighbours. There were always exceptions of course, but the purpose was to keep the nobles in line and out of open warfare with each other - a contrast to the chaos that marked 11th and 12th century France, for example.

The Marchers, meanwhile, were specifically given these rights from the outset. A ‘Marcher Lord’ was given a compact region of land from which they were expected to raise large retinues of soldiers with whom they were to war upon their neighbours - often fighting each other practically as much as the welsh. Marcher lords could raid, build castles, establish their own alliances, and were basically expected to do so as the Normans continued to drive further into Welsh areas. Often Norman marchers married into welsh noble families, allied with them, and there grew up a ‘grey area’ of nominally welsh lords who were under the jurisdiction of English barons - this would become a major issue in the late 1200s as Princes of Wales like Llywelyn claimed their loyalties, despite their legal status being somewhat debatable.

Many marchers were given even greater rights and privileges within their lands. The ‘County Palatine’ of Chester, practically the chief fiefdom of the Marcher Lords, meant rights that essentially made the controller a King in their own land - they controlled courts, judges, sheriffs, and more that made them almost independent. Perhaps unsurprisingly, the Earl Palatines were often involved in baronial trouble - Ranulf of Chester is a classic case during the Anarchy, for example.

In summary, the feudal system set up by the Normans limited the military capability of the King, in a way. Armies were expensive, taxes unpopular, and conflict on the Welsh frontier constant. Norman Kings relied on powerful, compact, militarily competent barons to defend and expand the Kingdom in Wales. These practically independent mini-realms of course created issues as time went on, and Marchers were at the center of countless rebellions, uprisings debates etc. Edward I, for example, relied on men like Roger Mortimer to provide troops for his victory over Montfort at Evesham. But Edward also began a process of weakening and legally consolidating the marchers into the wider Kingdom he ruled. With the earl of Gloucester, for example, who ruled a huge marcher lordship in Glamorgan, he essentially took his land away and granted it back under the understanding that it was no longer with the rights of a march. After the 1280s conquest of Wales by Edward, the welsh marches were far less important militarily - they continued to enjoy some residual rights however. Focus from the 1300s onwards was the Scottish Marches - a completely different but equally interesting story.

Hope some of this was helpful! I’ll add sources shortly when I’m off my phone.

Edit: here’s some sources!!

Barlow, the Feudal Kingdom of England, 1042-1216

Barlow is honestly my favorite for a comprehensive look at this period - his knowledge on the development of the justice systems is fascinating for example. He also does a great job going through the development of Norman and Angevin political systems and how they were shifts or continuities from the previous establishment.

Prestwich, Plantagenet England, 1225-1360 (2005)

Prestwich, Michael (1997). Edward I

Prestwich is the best when it comes to the Three Edwards - if you’re at all curious about the late 1200s, the second Barons rebellion, etc then please give these a read.

Trevelyan, G. M. (1953). History of England.

Tevelyan is a good general review.

Book, Christopher (1963) The Saxon and Norman Kings

This book is pretty short, but I appreciated it in this context for how it looked at the differences between the thegns and witan of the Anglo Saxons and the barons and Magnum Conciliam of the Normans.

The Welsh Wars of Edward I by John E. Morris

Morris’ is maybe more than any book what made me fascinated with this era of history. His laser focus on the documents from the era of Edward’s conquest is just impressive. But he also explains the conflict between Gloucester (Marcher Lord of Glamorgan) and Edward so poetically and conclusively that it’s indispensable for this topic in particular.

Enjoy!

I can compliment /u/cazador5's excellent answer on the Welsh marches by adding a little bit about the English marches against Scotland. This is a bit of rambling answer filled with whatever came to my head, so I'm sorry if it's a bit uneven or doesn't entirely answer your question; I'd be happy to expand on any points which aren't entirely clear.

Like the Welsh marches, the Anglo-Scottish border was a militarised frontier zone, heavily fortified with unique administrative and judicial structures in place which set them apart from elsewhere in the realms. This structures were formed in a long period of evolution and experimentation to suit the needs of border defence and administration: defending the realm from invasion in times of war, and dealing with cross-border conflict and tensions in times of peace.

Unlike the Welsh marches, which lost their military significance (if not their legal and administrative uniqueness) as Wales was gradually brought under the control of the English crown, the military defence of the northern marches remained of key significance throughout the 15th and 16th centuries, even after the two realms themselves settled into more peaceful relations. Borderers on either side of the line had to deal with the peace-time raiding of the 'riding surnames', and England remained fearful of potential invasion via its northern border.

The exact nature of the march evolved over the middle ages. Broadly speaking, the institutions we're talking about started to evolve from the reign of Edward I (reigned 1272 - 1307) onwards, as previously peaceful relations between England and Scotland were replaced by a more hostile relationship of alternating uneasy truces and open warfare. A 1249 treaty between Henry III and Alexander III of Scotland formalised many of their key aspects, but they continued to develop right up to the regnal union between the two realms in 1603.

Both realms divided their border region into west, middle, and east marches, and each of these marches was governed by a Lord Warden, an important administrative officer. In the medieval period the warden was invariably a high-ranking, resident aristocrat, usually drawn from the leading northern noble families.

As /r/cazador5 says regarding the Welsh marches, the marcher lords of the Scottish border were often powerful families with extensive estates and a degree of autonomy from the crown; their usefulness in the defence of the frontier depended on their manraed, or the armed service they could call upon from their own estates, and their ability to build and maintain fortifications. The office of Warden, when they held it, gave them additional authority and prestige, but this supplemented rather than replaced the natural authority they held as powerful regional magnates.

The clearest example of this is the Percy family, earls of Northumberland, who were enormously powerful in the region and had extensive landholdings throughout the north. Various earls of Northumberland held office as wardens, especially of the east march, which was where their territorial base was strongest, and their strong regional position made them influential on a national scale.

In the early modern period, when many of the northern families were suspected (or known to have) Catholic sympathies, there was some experimentation with office holders, including men of relatively humble standing, southern aristocrats, and, for a time, Henry VIII himself, but the status and the duties of the office remained largely unchanged.

The marches had their own legal system, the leges marchiarum. This included internal authority for the Wardens--the authority to try English individuals for crimes related to the frontier or the opposite realm, for instance 'march treason', or collaborating with Scottish incursions into England.

But the leges is more famous for its international aspect: 'days of truce', at which wardens from both realms met to hold a joint court, at which crime committed by subjects of one realm in the other could be tried and punished, and restitution could be agreed.

So this is one of the ways in which the designation of 'march' was significant to those living there: the imposition of this separate legal system, lying on top of the other legal systems which ran through medieval and early modern England.

Living in the march also imposed additional duties on landholders and tenants. These included the duty to muster for armed service in times of war, and contribute to border defence in times of peace through standing watches and attending upon the warden at days of truce.

Many of the tenants of the marches held their land by 'tenant right', which obliged them to maintain arms, armour, and horses for border service, and to muster at the command of their landlord or the prince. In return, such tenants were freed from many of the high rents, fines, and charges which landlords were accustomed to imposing on their tenantry.

The administration of estates in the borders was often different to what would be expected elsewhere in the realm, too, with the minor offices of the estates changed to suit their military demands. To use the Dacre estates in Cumberland as an example, the barony of Burgh (just across the Solway from Scotland) was administered by a 'martial steward', while Gilsland (ten miles or so from the border line) was placed under the charge of a 'land sergeant'. As well as the usual duties of estate-management, those who held such roles were expected to actively defend the local people, and lead them when called upon to do so by their landlord.

For much of the medieval period, the entire north of England was exempt from some taxes, on the grounds they they needed to be prepared for potential Scottish invasion. As the Scottish threat diminished in the 16th century efforts were made to raise new taxes elsewhere in the north of England, but the marches themselves were mostly exempt from these measures. The Elizabethan crown took on an active role in attempting to ensure that the tenantry of the borders were not subject to overly high rents and fines so that they would continue to be able to play a role in border defence; to an extent, this ensured that the landless people of the marches were protected from some of the extremes of the economic reforms elsewhere in England which were so detrimental to the poorest in society.