Military procurement has always been a bit messy, with conflicting goals of fiscal prudence, national readiness, empire-building within the military or government, profit-taking, and catering to key constituents. However, the specific idea of "R&D programs" which result in poor returns and public acrimony is mostly a modern phenomenon.

Procurement itself is a multi-faceted issue, so I'll stick to the R&D / innovation portion. Keep in mind that there was always a lot of money (relatively) involved in procurement of ordinary things: uniforms, food, tools, lumber and other raw materials. Public outcry over misuse of public funds or profiteering in this regard seems to be perennial.

With regards to America specifically, there is an inflection point after WWII when peacetime procurement and R&D goes from a relatively small process to an entire industry. It's no coincidence that Eisenhower spoke about the "military-industrial complex".

During the 19th century the standard models for military innovation were driven by either internal goals of the military, or the efforts of private inventors who then attempted to sell the military on some new innovation.

An example of the former would be the development of the Dahlgren naval guns. Young Lt John Dahlgren set out to design a modern, safer, high-powered naval gun as a direct result of the need he saw. He contracted with private firms using government funds to cast guns of his design, and oversaw the testing of these guns. Most of the guns were cast by private industry. Dahlgren received royalty payments on guns made with his designs.

An example of the latter would be the Spencer repeating rifle. Spencer pressed hard to get his invention adopted by the US Army, but was resisted by conservative forces within the Army and its Ordinance Department. It eventually became a very popular weapon within the Army and the US Navy.

In the second half of the 18th century, naval technology advanced rapidly. Prior to this a "battleship" could be expected to be a minor improvement over previous ones, and would serve alongside older similar ships for decades, maintenance allowing. But from 1860 onwards, there were several generations of warships that so outclassed the previous, that older ships were rapidly relegated to lesser duties, sold to second-tier naval powers, or scrapped. Because ships were so expensive, and deemed so crucial to national power, these design debates and procurements became matters of widespread peacetime interest. The processes themselves often remained highly technical, but the expenditures were debated in press and in legislatures. The advent of armored steam-propelled ships, breech-loading guns, the Dreadnought-era ships, each produced a wave of national doubt, pride, panic, and spending. Magazines and journals debated the value of battleships in the era of the self-propelled torpedo, or the battlecruiser concept. In the USA specifically, naval expenditure was seen in light of America's appearance on the world stage.

The design, though, was for the most part driven by military experts and executed by a combination of private and military builders. The main exceptions being some innovations in submarines, and propulsion. Innovations were designed, built, and tried. Adoption was slow and conservative. In this sense, "R&D" was still seen as a risk shared by the government and private inventors.

Innovation occurred in other areas, of course, but all of those areas seemed highlighted in naval applications. Larger artillery. Telegraphic communication. Wireless communication. Aerial observation.

During WWI, the American procurement effort centered in quantity, taking a small peacetime army and making it large and well-trained and supplied enough to fight in Europe.

During and after WWI the center of innovation changed to aviation. Rapid advancements in engine power-to-weight ratios, and material science, allowed aircraft to increase in speed, climb, and payload. As with battleships previously, new generation of aircraft would tend to eclipse older ones. Even at sea, aviation had a large influence. For the first time, the see governments and especially the USA begin to enter into ongoing partnerships with private industry for ongoing innovation.

Government subsidies, in the form of peacetime military contracts and also US Mail contracts for airmail, were crucial to the development of the domestic aircraft industry. Military requirements drove aircraft design. From the beginning there were objections and questions from taxpayers and competitors. There was a major scandal involving Air Mail contracts and the US Army Air Corps in the 1930s.

In the interwar period, the Air Corps pioneering procurement procedures such as permanent liaisons at manufacturing facilities, a more formal RFP process, contracts that included specification for delivery of spare parts over time, etc. For the most part these peacetime developments were seen as necessary, though as always "waste" was an issue, both real, and as a political football.

So, WWII. America becomes "the arsenal of democracy", and turns its massive and undamaged industry to military production. And in some ways, never fully stops.

When the war ends, the need for massive production of military goods ends. However, developments during the war suggest that an ongoing technological race exists, which exceeds the pace and scope of all previous ones. It is seen as crucial to maintain a pace of research and development sufficient to protect the nation in peacetime. Areas highlighted during or near the end of WWII include jet planes, advanced submarines, sonar, radar, encryption, rockets, and of course nuclear weapons.

Three things conspire to keep the military expenditures relatively high (although of course a huge drop occurs when the war ends).

First, the aforementioned technologies are all developing quickly, and are all seen as crucial.

Second, the "peace" turns "cold war", making the need for a larger peacetime armed forces.

Third, the more modern the weapons system, the less it seems wise to rely on making weapons and ammo and such just before or DURING a war. With the speed and range of postwar planes, and later missiles, and with mechanized armies, wars might last days or weeks. Complicated weapons mean you can't just ask the blacksmith to switch from horseshoes to muskets; it takes advanced factories to make a modern jet, and it takes ltoo long to make them once war starts. You need them BEFORE war happens.

The net result is the need to equip a relatively large peacetime army, with relatively NEW technology, refreshed often as tech develops. Private industry, with rare exceptions, isn't interested in making HUGE speculative expenditures. Instead, the R&D and procurement process becomes a permanent mode and an expensive one.

All of the old tensions remain. Divisions over dogma and policy, empire-building, por-barrel politics, private profit-taking, political posturing. But now the speed of the cycle and the expense means the public cares more, or is made to care more,

Technology becomes a public obsession, manifested in the race for the sound barrier, for space, the moon, the north pole (by nuclear sub!). So does failure of expensive weapons systems. When it takes a decade to prefect a new design, obsolescence is a constant risk. Costly dead-ends include the B-70, Star Wars, and the Sergeant York air defense system. Entire generations of weapons get built, barely used, and replaced at great cost because, you can't afford to NOT upgrade if you might be at war with the USSR next year. The cost is borne by the unrivaled American economy, and sales to other nations.

Again, R&D is only one part of procurement. Great amounts are spent on ordinary things like salaries, bases, medical coverage for soldiers and for veterans, etc. Reducing the number of bases from the WWII peak to today, was a decades-long effort pitting cost savings against "local economy" advocates. Reagan's 600-ship navy was as much an exercise in economic warfare against the USSR, as it was a true military expense. The cost of OPERATIONS is huge, in both lives and money. A peacetime force, even with lots of training, is way cheaper than the price of going to war, especially the kind of long-distance wars the USA has fought since 1917. The political decisions are huge. The procurement efforts for all the boring stuff is also huge. It's just seldom highlighted.