The academic consensus that I was raised in was that this was due to a central trifecta- the interaction of superscience, HYDRA, and the Cold War. Operation Paperclip is what first brought ex-HYDRA scientists into the US heartland, and it began because of the immediate need to develop American based superscientific weaponry upon the war's completion- everyone feared an immediate new war with the USSR, after their occupation of large swathes of Europe. HYDRA's work was the most weapons-based superscience division of the war, which is what attracted the USA to its importation in the form of both men and materiel. These continued to form the backbone of the superscientific line of defense of the USA throughout the Cold War.

But why was the growth of HYDRA within SHIELD not properly noticed? There is a very clear reason behind that; even though HYDRA had split from the Nazi regime and become its own quasi-polity, a fact which was not made public to the American people for quite some time, they were still fundamentally considered to be synonymous with Hitler's regime and the Nazi party. Both this original HYDRA and its Nazi cousins were decidedly unsubtle about their imagery, ideology, and authoritarianism. This, in combination with the USSR's often colourful superscience, led to a false association between these two things- essentially, that a superscience based organisation not centered around democratic ideology would always look and sound as unsubtle as the original HYDRA.

The surviving HYDRA members who were interested in continuing with their goals learned from this, and an ideological shift occurred; instead of upright, stolid, unshakeable cult they instead became a network of shadows, connections, and ideologically flexible. Their recruitment was based upon being accommodating enough to be attractive to any valuable target for membership. As time wore on, many HYDRA members were increasingly unconcerned or indifferent to the organisation's history, putting no real stock in the Red Skull or the original HYDRA, and actively considered themselves a new, better breed of HYDRA. Neither did they remain meek- the 20th century was a secret battleground between HYDRA directed ideological manipulation of American culture and society, and those who resisted it, though those who did so did not realise what it was they were fighting. It is from the 80s onwards that HYDRA gained a relatively firm direction over American society, enough to promote values friendly to theirs, and to create environments conducive to recruitment. It is from this period that mass-membership of HYDRA began, and given that the 80s saw a newly aggressive phase of the Cold War they had every reason to be camoflaged from observation.

However, nobody is invincible, and it was only by murdering Howard Stark in 1991 that HYDRA was not discovered in the immediate years after the Cold War ended.