I mean the short answer to this is the same way we get a lot of borders: culture and politics and treaties and relatively arbitrary agreements about landholding.

It's also really complicated because of the presence of Romans, Gaels, Picts, Angles, Saxons, Jutes and eventually the Normans at various times across ... oh ... 1200 years before we finally start to see more concrete borders which approximate the current state of affairs.

The first thing we have to remember is the Antonine Wall and Hadrian's Wall were begun pre-150 and these were built by Romans who were around far before there was even a concept of Alba or Scotland and England, so these were never meant to mark a distinction between the two. Mostly it was to mark a boundary between the northern peoples and the various groups of Britons who were under Roman governance. (If you want to know more about this period, you should see James E. Fraser, From Caledonia to Pictland: Scotland to 795, 2009).

The first real border we see here is about tenth-century. Prior to the ninth century is outwith my area of confidence but around the early tenth century, we see a border emerging between the Gaelic and Pictish kingdom of Alba and that of Northumbria (this is about a century and a half after Kenneth McAlpin who united the Gaels of the northwest and the Picts of the northeast). So Alba / Northumbria was the border at this time and it was drawn roughly from the Firth of Forth through to the western side of Scotland.

This was a combination cultural, social and political border based on the ethnographic and cultural makeup of the two societies. To bring in the linguistic angle: the peoples of the kingdom of Alba would have been primarily Gaelic-speakers at this point in time, while those on the "English" side of the border were speaking Cumbric and eventually some Old English.

This border persists for a while, but up until about the thirteenth century, various attempts by Scottish kings to push southwards and English kings to reclaim this territory resulted in a lot of "official" back and forth. That's partly why the Anglo-Scottish border throughout the later medieval and early modern period had such a mix of English and Scottish culture/surnames/cross-border crime. Important dates:

• 1018 - border now lies along River Tweed after the Lothian region is conquered by the Scots at Carham.
• 1237 - Treaty of York designates the new border as Solway-Tweed

And it basically stays there. There are some minor adjustments based on agreements set out by military conquests by England (such as Berwick-upon-Tweed - 1482). Several political developments (Treaty of Union, 1706; Act of Union, 1707; Berwick annexation, 1746) really helped to resolve the issues of lawlessness created by the problematic administration of march or border law on either side of the border by introducing closer and more organised oversight. So this is the period when the distinction becomes even clearer from an administrative point of view.

To get back to the linguistics. In the twelfth and thirteenth centuries, there was a great deal of Anglo-Norman influence on the Scottish kings seated in the Lothian area. I speak more about this in an older answer I gave last year, but the general idea is that the French-speaking Normans were considered more "sophisticated" than the Gaels and it became politically and socially prudent to shift away from Gaelic and to embrace Anglo-Norman/French culture and language.

---

For the border stuff you want to see primarily:

Woolf, Alex. From Pictland to Alba, 789-1010 (New Edinburgh History of Scotland). Edinburgh: Edinburgh University Press, 2007.

Oram, Richard. Domination and Lordship: Scotland 1070-1230 (New Edinburgh History of Scotland). Edinburgh: Edinburgh University Press, 2011).

For sources on the linguistic shifts, see the previous answer linked above.