If Wikipedia is correct in that surnames took until ~1400 to be common among the English, then I have an answer.

French was still the lingua franca, bad pun intended, among the elite for 3 centuries while surnames were beginning to catch on. You would have to pick de / d' if you were well-to-do, and it is likely the relatively elite who would take up the custom of surnames first, then the middling merchant class, then the peasantry. Even the word 'surname' came from Frenchmen.

French was the language of Parliament and the courts until about that time. The Pleading in English Act of 1362 forced the courts et al. to speak in English, and record in Latin, replacing French. The first king to take his oath in English and not French was Henry IV (1399-1413). His son Henry 5.0 was the first to be able to actually write in English (all of this since 1066 of course.)

You might be interested in this thread from a few months ago, which has quite a bit of discussion on it.

If you look at noble styles (that is, modes of address) you'll see that the correct way to refer to aristocracy does in fact use "of". E.g. Robert Alexander Lindsay, 29th Earl of Crawford and 12th Earl of Balcarres.

Even more extreme, those with the titular dignity of prince don't really use surnames except where they have to adopt one for convenience. Instead they just have their forenames and their titles. For example, Prince William's full title is:

His Royal Highness Prince William Arthur Philip Louis, Duke of Cambridge, Earl of Strathearn, Baron Carrickfergus, Royal Knight Companion of the Most Noble Order of the Garter, Knight of the Most Ancient and Most Noble Order of the Thistle, Personal Aide-de-Camp to Her Majesty The Queen.

A lot of "of" in there, and no surname.

(Of course all of this is referring to people in third person. Addressing them directly you'd just say "my lord", "your highness", "sir", etc.).