I can't speak in depth to the earlier history of Saudi-U.S. relations, but I can at least address the more general question to some extent. Regarding the idea that the U.S. and Saudi Arabia being the top two oil producers might lead to competition instead of collaboration, that would make sense in a situation where both countries were solely suppliers selling to another party or parties. However, while the U.S. and Saudi Arabia are indeed the top two producers of oil (with Russia close behind Saudi Arabia), the US is also the top consumer of oil and consumes more than it produces -in recent history it held an even larger share of the consumption than it does now ( US Energy Information Administration as well as the World Population Review ). Meanwhile, Saudi Arabia consumes only a portion of what it produces, meaning that it is free to export the rest to countries like the US. Furthermore, the US does not simply consume all of the oil it produces and then purchase only the 2 million barrel/day deficit from the global market afterwards, so the reliance on international oil prices and trade ties is somewhat greater than the import deficit would indicate.

So what that means is that despite being competitors in a sense, the US is also reliant on maintaining good relations with other oil producers in order to supply its needs. If you look at the other top ten oil producers, you also see countries such as Russia, China, Iraq, and Iran, with whom the US has not always been on the best of terms within the past 50 years, so it makes sense find trading partners when and where you can. Instability in the USSR, Iraq, and Iran during that period have also contributed to some concerns regarding reliable oil production from each.

To understand the effect of poor relations with Saudi Arabia or other oil producing countries on our economy from a historical perspective, the 1973 oil embargo by OAPEC (the arab majority of OPEC) and resulting crisis provides a powerful example. The embargo was targeted at supporters of Israel in the Arab-Israeli war, including the U.S., and international oil prices quadrupled. Despite the fact that as late as 1970 the US had a slight oil surplus, the embargo heavily strained the US economy and caused other US allies to rely on US assistance in obtaining energy supplies as well. Combined with other factors, this created stagflation in the United States, and though the embargo is considered by some to have been a failure in terms of winning the war for the Saudis, it cost the US heavily ( US Office of the Historian )

The influence of economic factors in politics can't really be understated, so despite many conflicts in values and policy between the US and Saudi Arabia, the importance of Saudi oil has led to a partial dependence on maintaining good relations with the Saudis. Based on past precedent, the US will likely continue to have that pressure point and that particular contrast of western values versus foreign policy actions for as long as we rely on oil as a primary energy source.

I could send you the entirety of my research on this topic, but I'll try to be brief and as comprehensible as possible in my explanation of the origins of the US-Saudi alliance. If you have any additional questions, feel free to let me know, I'd be happy to answer.

Here we go

King 'Abd al- 'Aziz' of Saudi Arabia was made aware of the potential wealth lying beneath his feet after oil was found in Bahrain, by his personal friend Harry St. John Philby, a British Arabist, traveller and merchant in the kingdom who converted to Islam under the tutelage of 'Abd al- 'Aziz. Standard Oil of California showed interest in a Saudi concession at al-Hasa and the company was encouraged by its recent findings in Bahrein. By the end of May 1933 a deal was reached and signed. Socal had to make an initial loan of £30.000 in gold or its equivalent that was to be paid in advance along with £5.000 for the first year's royalty. If the agreement lasted for 18 months another sum of £20.000 was to be loaned. The loans were not reimbursable and had to be recovered by deductions of royalty payments. If oil was discovered on the 360.000 square miles of the allocated territory, Socal would pay an extra £100.000 if it was found in commercial quantities. The company would also not interfere with the internal workings of the kingdom. Finally, Socal would get preferential treatment to acquire other concessions if it could match other offers made to the king. The deal was to last for a period of 66 years. As usual, the concession was held under a subsidiary company of Socal; The California-Arabia Standard Oil Company (CASOC), which would later be known as Aramco.

With Socal acquiring the first concession in Saudi Arabia, another problem came knocking at the door of the oil company in San Francisco. With the Bahrein findings of 1932, Socal built a refinery four years later for transforming crude oil into commercial products (gasoline, diesel, kerosene,…). However, Socal did not have any markets or transportation facilities in the Asia-Pacific; it had to look for someone to do this for them. Initially, Socony marketed all of Socal's products in the Far East, but that market was closed to the California-based company with the self-denying clause of the Red Line Agreement Socony was bound to (The Red Line agreement is a complex Middle Eastern oil agreement that would shape the future distribution of Middle Eastern oil reserves). Texaco on the other hand was not bound by the Red Line Agreement, and although the company exported its Texas oil to the Far East, it had no refinery or commercial oil wells in the region. The merger was therefore beneficial to both parties, and so in 1936 Texaco bought 50% of the CASOC concession of Saudi Arabia and Bapco for $21 million. The joint-venture became known as the California-Texas Company (Caltex). In exchange, Socal would get access to all of Texaco's market facilities East of Suez. I

Oil was struck in Saudi Arabia in March 1938 at Dammam. Companies from all over the world rushed to 'Abd al- 'Aziz to get a concession from him, even Germany and Japan. But CASOC, with its preferential treatment clause got the winning bid.

Although the discovery of commercial quantities of oil made the financial burden of the kingdom less heavy, the outbreak of the Second World War brought new economic woes to Saudi Arabia. Pilgrimage revenues tumbled at the outbreak of the war in 1939, and pilgrimage from India and the Dutch East Indies, which brought the considerable revenue sources, was cut off two years later. And with the amount of pilgrims entering the country in decline, so too were customs revenues. In addition there were also crop failures and the money 'Abd al- 'Aziz owed to tribal leaders for their loyalty along with the cost of imported vital needs increased. The king pressured the oil companies, British (IPC) and Americans (CASOC) alike for more money, but the Americans were not sure they had the necessary funds to come to the king's aid. It was in this context that CASOC executives approached James A. Moffet, the former chairman of the board of Caltex and who was close to President Roosevelt, to ask for government aid. In exchange for an annual sum of $5 or $6 million for a period of five years, the US government would get access to oil far below the market price. Moffet made it clear in a memorandum to Roosevelt that oil was not the reason why the agreement should be backed by Washington, why would Washington be worried about petroleum when the United States were pumping more than 3.7 million barrels per day in 1940 compared to Saudi Arabia's twenty thousand? it had more to do with the fact that the king was pro-ally but more importantly because "No other man in the Arab countries, nor among Moslems the world over, commands prestige equal to his." This refers to the house of Saud being in control of the holy cities of Mecca and Medina. CASOC executives on the other hand believed that there were also political gains behind the "Moffet deal" to counter British interests in the region, though this is still a debate amongst academics.

Though Roosevelt and his administration were initially in favour of the deal, there was no authority to loan out the required $5 to $6 million to Saudi Arabia, added to that, Lend-Lease was only authorized by Congress to "democratic allies" and Saudi Arabia was by no means a democracy. Hull and Roosevelt hoped that the British could take care of the matter, with the President making clear that 'Abd al- 'Aziz' problems were "a little far afield for us!" Roosevelt folded on the "Moffet deal". It is important to remember that in the summer of 1941 the United States were still neutral in the war, Roosevelt was afraid of the political repercussions an agreement with Saudi Arabia might have at home, particularly a reaction from the isolationists. Nevertheless, an agricultural mission was sent to the country along with an official permanent representative in Jeddah in 1940, something the companies had urged Washington to do long before.

Washington's position changed just before America's entry into the war with the findings of Secretary of the Interior, Harold Ickes. In October 1941, Ickes made a study which concluded that there will come a time in the near future where the proven domestic reserves of the United States will not be able to provide the necessary oil quantities of domestic consumption. The United States were destined to become a net importer of oil in the near future. In addition the Truman Committee estimated that in 1944 the United States had produced 70% of the oil used for the allied war effort. The fear of American oil shortage after the war was real. Oil needed to be found abroad; luckily the United States had CASOC in the Middle-East with a concession that was proven to be more promising before the war began. While the war raged on in Europe, it became clear that Washington would turn to Saudi Arabia and the Middle East with its considerable potential oil reserves, which in 1943 were estimated to be more than 56 million barrels. That number rose to 100 billion barrels for Saudi Arabia alone after the DeGolyer expedition.

A more important reason as to why Washington looked to the Middle-Eastern oil fields was transportation. With U-boats harassing American transport ships in the Atlantic to help the war effort in Europe, the United States were urgently in need of oil reserves closer to the military scene in Europe to reduce the weight and pressure of the Western Hemisphere reserves of the United States, Mexico and Venezuela. The latter was recognized as an area of strategic importance to the United States, since Venezuelan oil could be used to supplement the exports and American reserves as well.

By the end of 1943, the Roosevelt administration's interest in Saudi Arabia grew more whenever Washington was made aware of the uncertain future of American oil quantities. Pushed by the false British representations of the oil companies established in the region, the administration feared that Saudi and Middle Eastern concessions might fall in the hands of the British. The Roosevelt policy on Saudi oil took shape in the form of three distinct approaches. The first took the example of the British government shareholding in Anglo-Iranian, which pointed to a direct involvement by the American government in the Saudi concession and a solidification of the relationship. This took shape with the creation of the Petroleum Reserve Corporation (PRC), the main objective of which was to obtain oil reserves abroad. This option was favoured by Secretary of the Interior Harold Ickes, who created the PRC, and Roosevelt. The second option was to work out a plan with the British to ensure that Middle Eastern oil would be equally distributed. This took form in the Anglo-American Petroleum Agreement of March 1944. This road was favoured by Secretary of State Hull and the big oil companies, but not the smaller independents, which were backed by Congress. Finally, the last option was to give free rein to private enterprise with political backing from Washington, who would ensure equitable access to foreign reserves and guard American enterprise overseas from harm. The latter option was eventually agreed upon after the two formers failed.