In short, this was a gradual process that spread across the early modern period.

For context, Scots descends from Northumbrian Middle English, with parts of what are now the Scottish Lowlands having been part of Northumbria from early on, and continuing to be seen as ethnically English even after incorporation into Scotland - what Adam of Dryburgh called ‘the land of the English in the Kingdom of the Scot’s’. De Situ Albanie, combining a mythical history of Scotland (or ‘Alba’) with geographic and cultural descriptions, describes the Firth as the boundary between the English and Scots.

For the medieval period it was largely called Inglis (English) or similar variants, and something of a lingua franca in the towns, which were largely bases of trade, which was largely with the much larger England. Many merchants from Northern England and their families settled throughout the Middle Ages in that context. It took a few centuries for the inevitable forces of language change to lead to a recognisably distinct Scottish dialect, but this started to be acknowledged in the 15th century, when writers refer to the language as ‘Inglis’ when distinguishing it from ‘Erse’ (‘Irish’, or rather Scots Gaelic) within Scotland, but increasingly as ‘Scottis’ when distinguishing it from English in England.

One major force started in the late 15th century: the consolidation of a chancery-based ‘standard’ English, based on the variants around London, took over and formed the basis of modern English. Before that, Middle English had been very pluricentric and drawing a distinction between Scotland and the northern English dialects was not so clear. But with the increasing standardisation of English around the court and official documents, Scotland was both influenced by it and the distinction between the dialect in Scotland was made clearer. Edinburgh was seen as the capital of Scotland by the 14th century (Jean Froissart refers to it as such), and was its principal and most influential city, and the Scottish court largely spoke Scots, rather than Gaelic. Over the early modern period, amid conflict with England, the desire to keep the Scottish identity separate led to literature in ‘Scottis’ by its own ‘makaris’ (poets) increasingly separated from standard English forms, and by the time of Shakespeare and Ben Jonson it is clear that the Scottish variety of speech was considered distinct: if not necessarily a separate language, they had their own Scottish ‘language’ (Gaelic aside). As it continues to diverge, it was widely referred to as a separate language by the 18th century. Even then, and even now, this opinion isn’t entirely universal in Scotland, but is of course in large part a political question. One of the most influential people to emphasise its distinctness was of course Robbie Burns, by the late 18th c.

Another force was the development of Scottish English, as Scots speakers and others learnt the modern English standard, with Scots losing speakers by this shift. Especially upon the union of the crowns, when the Scottish monarchy and court merged with the English one, and even further with actual Union, Scottish English (standard English with largely Scots influence) developed and would have made calling ‘proper’ Scots ‘English’ very confusing, as it would be more likely to contrast it with what would be seen more as an ‘actually’ English language, one that had now diverged a great deal, than with Gaelic.

Even in the last few decades, polls in Scotland show a fairly split attitude on this question, but as Scottish nationalism and independence movements have grown in prominence this has increasingly been skewing towards the status of a separate ‘Scots’ language.

Of course, linguists don’t generally have a standard for the difference between a language and a dialect, so the question is ‘academic’ (or rather, precisely not academic!). But when we want to refer to that speech variety, ‘Scots’ is now the name of choice.

An aside: Likewise, by the way, Scots Gaelic was largely called ‘Erse’, or Irish, as it had come over largely around the fifth to seventh century or so with the kingdom of Dál Riata, though the exact process of Gaelicisation are murky. With the Brythonic Cumbric spoken in early Strathclyde and the ‘original’ Picts, who were absorbed into Gaelic identity across the early medieval period, all four countries had some linguistic representation!