This question may be more suited for /r/linguistics, try there too.

it never caught on widely

Simple answer: It just never really caught on widely.

Longer answer: We can see in things like the Last Will and Testament of Alfred the Great that neither Old English nor Middle English used accents, or at least didn't use them consistently enough that you could leave them out entirely and it be okay. People did use a series of markings in Latin that acted as scribal abbreviations (more examples here) - like m8 for "mate" but Latin and completely acceptable in texts. These look like accents/diacritics, but aren't, or at least aren't yet.

Diacritics developed from this practice. A little letter was sometimes put above or below the "main" letter - in German, ü developed from a u with a little e on top and was stylised over time into the two dots. This just didn't really happen as much in English, although you do see Latin abbreviations in English names in old wills (which were written in Latin - I saw this on a paleography course and can't find any examples now but will add later if I can).

The introduction of the printing press caused a certain amount of "flattening" of the letters in English - older English letters like þ disappeared because the presses themselves came from other countries that didn't use them, so they literally didn't have the means to print them. It's likely that this cemented the idea that diacritics/accents weren't really necessary as they were not only rare and inconsistently used, but now they were also hard to print using available equipment.

Modern diacritics in French, for example, were standardised in the 18th century - before that they were similarly inconsistent in use but went the other way with it and formalised the system, rather than letting it disappear.