I can't speak for every user, but the US Army Light Rifle trials after WW2 and the similar trials performed by Britain and the rest of NATO point to a fair amount of problems with optical sights for general issue. During these trials, the British were evaluating a series of rifles that were fitted with 1x power scopes on top of a carry handle (the EM-2 being the most well known of these) and NATO as a whole was evaluating the FN FAL/T48, which included the development of a special dust cover with an optical sight integrated into it.

Several major issues were found over the course of trials with these optical sights. Traditional scopes and optical sights (including the ones being evaluated by NATO at the time suffered from relatively narrow fields of view - a major liability for soldiers that may have to engage at very close ranges. Durability of optics has always been an issue, and their size and complexity generally makes them significantly easier for soldiers to damage than traditional iron sights. On the FAL in particular, mounting of optics was an issue, as the most obvious location to mount them - the dust cover - is a relatively loose mount that's poorly suited to mounting optics without some kind of special modification. For the British bullpup projects at the time, durability of optics was a major issue owing to the lack of backup iron sights should the optics break. The issue of cost also played a part - the Light Rifle trials found that the benefit of optics like those on the EM-2 over bare iron sights wasn't worth the added costs associated with adopting a comparatively complicated sights for general issue.

Taking a wider look at things, there's the larger issue of existing guns not being suited to mounting optics. Looking at sniper rifles in WW1 - arguably the first war where optics on rifles saw fairly wide use - we see that scopes in traditional mountings like we're used to with hunting rifles today got in the way of things. Straight bolt handles that were common at the time got in the way of a traditional scope mount, stripper clip guides were obscured and made useless on the guns that had them, and en-bloc clip-fed guns were completely unusuable with a scope directly over the action. Mountings had to be designed for guns that had never been designed for such, and often times guns had to be altered to fit a scope. Scopes mounted off-center are pretty common in a lot of WW1 and WW2 era sniper rifles, and more famous ones like the Mosin 91/30 sniper models had their functional rate of fire significantly reduced because they could no longer clip-feed the magazine.

Serious looks at general-issue optics didn't really seem to appear until after WW2 (see the NATO Light Rifle Trials) and well into the Cold War every major power had adopted rifles that weren't necessarily designed with general-issue optics in mind. That made adoption of a general-issue optic a major problem - any general-issue scope had to be something that could mount on existing rifles, which could require significant modifications. Guns like the AK-47 and its derivatives particular presented a major hurdle - the loose mounting of the dust cover forced designers to go with a rail mounted on the side of the receiver, which in turn meant that any optics designed for AK rails are fairly bulky. Compare the PSO-1 scope of the SVD to a contemporary hunting scope of similar power to get an idea of how bulky the system is. The M-16 fared better in the long run, but only after a major redesign of the upper receiver to replace the carry handle and rear sight with a rail mounting system for optics and other attachments.