Counterinsurgency has not failed throughout history nor is it one single doctrine. Counterinsurgency is a very generalized term for different strategies in dealing with insurgencies, very much like the term conventional warfare when it comes to "regular" warfare.

The current debates surrounding counterinsurgency is based around what I deem to be a historically untruthful claim that the "myth" of counterinsurgency led to the bad results on Iraq and Afghanistan. This is a far more presentist view than a historian would accept and is false.

Your question should in this case be: Why has US counterinsurgency failed? What led to the failures in the counterinsurgency conflicts in which the US has been involved in since the 1960s?

I've written some posts on the topic before, but I am open to any questions you might have! Unfortunately, since your question does concern modern day practices of COIN, I'd have to remove it (but you can still reach me).

To what extent was counterinsurgency actually successful in Vietnam?

What has characterized past, successful counterinsurgency operations? Second part of this answer deals with how COIN was dealt with after Vietnam in the US Army.