English basically does have special letters. Italian, for example, only has 21 letters in the alphabet. English has 26.
The Romans used just 23 letters to write Latin: There were no lower case letters at first, and K, Y and Z used only for writing words of Greek origin. The letters J, U and W were added to the alphabet at a later stage to write languages other than Latin.
https://www.omniglot.com/writing/latin.htm
It’s always much more difficult to address a "why not” than to answer a “why” in historical linguistics, where so terribly many factors are at play. In this context, it is especially difficult because “special letters” is not a well-defined category.
Nonetheless, I will say that the principal reason modern English orthography does not contain characters native to English but foreign to what we think of as the “Latin” alphabet (in the modern typographical rather than historical sense) is, broadly speaking, the immense influence of French on the language during the Middle English and to a lesser extent Early Modern English period. There’s an argument for some minor Norse influence in this respect as well. For example, there’s the Ormulum as an example of Norse innovation, in English spelling.
By the end of the Middle English period, we had lost Thorn (Þ), Eth (Ð), Wynn (Ƿ), Ash (Æ) and gained then lost Yogh (ȝ) as well. Various scribal shorthand innovations native to England likewise failed to survive the period. These had lingered to an extent in Middle English itself, but ultimately, movable type was the final nail in the coffin for surviving Old English character usage and scribal abbreviations.
As for more recent departures from the standard orthography, the distinction between initial/medial (ſ) and terminal (s) ‘s’ is an orthographic habit which survived into Modern English, yet died in later modernity. Again, this is largely a function of the influence of movable type on written language. Wherein special cases (fairly literally), regionalism, and unnecessary distinctions were inclined to see themselves disappear. Particularly in regions influenced by or dependent on foreign typefounding.
First it is entirely relative of what constitutes a special letter and second from another perspective English is actually "missing" letters for the forty or so distinct sounds that can be made and most people are not taught digraphs (sets of letters that make a particular sound like "ch") diacritic marks to distinguish them.
For example English eventually evolved a distinction between o, u, v and w. In Arabic, these are all the same letter "و". Russian doesn't have "w".
Then in English we have a few different ways the letter "a" or "e" can be pronounced and foreign words (French, Greek, Latin) which are almost half of English follow an entirely different rule set. From a literacy perspective, having fewer letter isn't necessarily easier when you have to memorize exceptions.
Since the answers currently don't address this specifically, and since I'm guessing this might be what the asker was interested in: was there any specific reason that letters with diacritics have not survived in English outside loanwords.
French has a whole bunch of accents, German and Scandinavian languages have umlauts and so on. In some languages these count as their own letters (not as alternative versions of the 'base letter'). What led to these other languages gaining them when English didn't, or led to English losing them when others did not? It seems the answer might be quite different to the answer posted by /u/Yst about the Old English letters which are also missing from other languages (except Icelandic, and æ in Scandinavia).
