Isn't Gaelic an Irish language? And aren't Celts Welsh?

Modern English? After the act of union in 1707. The Urban elite shifted from Scots to English as the prestige language and other groups of Scots followed them over time to various extents.

The shift from Gaelic is older and more complicated. You've got to remember that the Scottish boarder has shifted south over the centuries with the Kingdom of Strathclyde and bits of Northumbria making up the southern areas modern Scotland. These southern areas probably never spoke Gaelic (the Celtic language in that area would be Cumbric). These areas would have spoken an early form of english which due to adoption by the urban elite spread north from the 13th century or so. At the same time it split off from what we might consider mainline English.

The answer will depend on what you mean by English and what region of Scotland you are talking about.

It's important to note that Gaelic was introduced to Scotland in the Dalriada region by settlers who arrived from Ireland prior to the 6th century CE. As Gaelic families intermarried with the Pictish people, the language spread until Gaelic was the predominate language in northern and western Scotland (10th c.). Further south, Old English and Cumbric would have been more common.

As /u/geniice wrote, because the border between Scotland and England shifted with some frequency, you would find variation in the Scottish Lowlands and Borders depending on time and place.

Although Gaelic did eventually spread southward, it took root in a more piecemeal fashion the further south you go and really only became widespread in Galloway. The spread southward essentially stopped when Malcolm III married outside of the Gaelic-speaking world and brought an Anglo-Saxon princess, Margaret, to Scotland. She gave her children names like Edmund, Edgar, David and Matilda and introduced a lot of English cultural elements to the Scottish court. This sort of arrested the spread of Gaelic and marked a turning point for the Scottish court, linguistically and culturally.

So, to answer part of your question, there were areas in southern Scotland that were never actually Gaelic-speaking to begin with.

As far as when Scots stopped speaking Gaelic, for some the answer is 'never'. Although Scottish Gaelic is an endangered language, there are still small communities of Gaelic speakers throughout the Highlands and Islands to this day.

For others, the shift occurred gradually between the eleventh and fifteenth centuries. It's important to note that the language of the church and courts in central and late medieval Scotland was Latin and (after a rather large settlement of Anglo-Norman lords in the twelfth-century) that the language of the elite was French (see the reign of David I). One effect of David's efforts to bring Scottish government and religious worship in line with the rest of continental Europe was to cast the Gaelic language and culture as sort of backwards or barbaric. This attitude trickled down among Scotland's nobility and many previously Gaelic-speaking families started speaking Norman French and Scots (Scottish dialect of English) in an effort to elevate themselves and fit in.

By the fifteenth century, Gaelic essentially vanished from central and eastern Scotland although it remained prevalent in the Highlands and in Galloway. It should be noted that these regions were more sparsely populated compared with the Central Lowlands, so although there was still a large area in which Gaelic was spoken, the population ratio was skewed in favour of English(Scots).

The remaining Gaelic speakers would come under attack in the eighteenth century as a result of Highland clan involvement in the Jacobite risings. In this period, various acts enforced by the Crown shook up social organisation in the Highlands. Gaelic-speaking lords and men who were important figures in the risings were transported or executed; whole communities were evicted due to enclosure... The language and culture really struggled from this point forward.

So the short version is, most Scots stopped speaking Gaelic sometime during the fourteenth century because it was considered more "sophisticated" to speak Norman French or Scots. Eventually Gaelic just faded away because it was more convenient and useful to speak what all the important and noble people (and marriage prospects!) were speaking.

Further reading:

Barrow, G. W. S. Kingship and Unity in Scotland, 1000-1306. 2nd edition. Edinburgh Classic Editions. Edinburgh: University of Edinburgh Press, 2015.

Bartlett, Robert. The Making of Europe: Conquest, Colonization and Cultural Change, 950-1350. Princeton: Princeton University Press, 1994.

Davies, R. R. Domination and Conquest: The Experience of Ireland, Scotland and Wales, 1100-1300. Cambridge: Cambridge University Press, 1990.

Davies, R. R. The First English Empire: Power and Identities in the British Isles, 1093-1343. Oxford: Oxford University Press, 2000.

Jones, Charles. The Edinburgh History of the Scots Language. Edinburgh: Edinburgh University Press, 1997.

Withers, Charles W. J. Gaelic in Scotland, 1698-1981. Edinburgh: John Donald, 1984.