America does not have a military base in "every country".

Anyway, being specific, probably the leaders for military deployments outside their home country in the modern era are the UK, France and Russia.

During the Cold War era the USSR did its best to try to establish military bases outside the USSR and its Warsaw Pact satellites in order to counter the USA's global network of military bases. This wasn't wholly successful, but significant Soviet forces were based at various times in Cuba, Syria, Yemen and Vietnam.

The UK's overseas deployments shrunk during the dissolution of its Empire, but post-Imperial diplomacy along with collaboration with the USA allowed the UK to keep a lot of foreign military bases. These include having significant forces in Belize, Cyprus, Brunei, Singapore, Oman, and Canada.

France, like the UK, managed to negotiate its drawdown of Empire into maintaining a significant military presence in many of its former colonies, although unlike the UK French overseas bases were mostly limited to Africa. But France maintained significant overseas deployments in a great number of former colonies, including but not limited to Djibouti, Cote D'Ivoire, the C.A.R. and Senegal, as well as in the former Belgian colony of Zaire.

Both France and the UK kept significant military forces in Germany during the Cold War, too, although I'm not sure that's what you had in mind.

Your initial though may be wrong, the US has not got a base in every country.

Going back in time you can start considering as a base the so called "coal stations", specially in the second half of the 19th century the steam became prevalent in the world navies, the US was not exception. This meant that the new vessels used boilers that needed coal and you can only transport a limited quantity of coal on board, hence you plan your tour or mission based on this. In order to project your naval power you need these coal stations to refill your coal supplies and keep going. Guantanamo is an earlier example of this, or Diego Garcia in the Indian Ocean for the British side. These would be early bases for me.

The bases you are really talking about are probably those of the Cold War and NATO era. The US was one of the two superpowers and has the world divided in commands (military speaking) hence US military presence is required in different degrees. The Cold War polarized international politics and made many countries choose sides in return for support and potential protection in case of conflict. One of the basic NATO policies established that should any member of the Treaty should be attacked all will be considered attacked prompting a common response, typically a military one. This was a message to the Soviet Union, if you attack Germany for instance, the whole NATO would response not just Germany. so depending on the strategic importance of the country, their local governments and the US needs you will see a huge degree of variations in terms of military presence.

Germany was the center of the hypothetical war theater should a NATO-Warsaw Pact confrontation have happened. Hence the main American Army assets and good part of the US Air Force in Europe landed there with West German (German Federal Republic) agreement. Army units (armored division, infantry, artillery etc) and fighter and fighter bomber and advanced control units to both contain the Soviet roll and try to obtain air superiority.

UK on the contrary followed a different path, being more distant from the main continental theater the Strategic Command saw it more fit to place there strategic bombers.

Turkey on the other hand went a different way, while the overall American military presence was reduced compared to Germany it had important strategic considerations, sharing a border with old Soviet Union Territories gave the US a chance to place there medium range nuclear missiles for instance (Jupiter) and eventually flew from there a good number of U-2 missions.

The Navy had different considerations, used a large base in Southern Spain (Rota) somehow the descendant of those old coaling stations, that is still used today by the US Navy. It gave an important logistics base to the Mediterranean Fleet and could have helped "locking" the Mediterranean Sea access to the Atlantic Ocean should it has been necessary. The base would there for decades since the sixties although Spain would not become a NATO member until 1982.

Other countries were not part of the NATO either but they aligned with the US and Western Europe in the global struggle , examples are Australia and New Zealand that had a separated Defensive Pact with the US (ANZUS if my memory is correct).

Nowadays the geopolitical situation has changed significantly, the US presence in Western Europe has been greatly reduced and has increased in other areas of the globe such as the Middle East on in the case of Europe pushed further to the East in new NATO members like Poland, Hungary or Romania, at any rate this presence is still far smaller than that of Cold War times. A significant US presence is still maintained in South Korea, you probably know the situation there, although this is not new as the Korean War was "halted" in the late fifties and the stalemate still persist. Taiwan is another hotspot but I am unsure about the US presence, their military certainly trained in the US much with US equipment specially the Air Force.

So bottom line in retrospective if you look back at the changing geopolitical situation the fact that the US has got such a global presence is not strange, other countries had no chance but seek assistance since their own armed forces would not be sufficient to deal with the threat. The Soviet Union (now Russia) and China have tried the same at lower scale trying to have mainly naval bases in Africa and Myanmar for instance, during the Cold War the Soviet Union kept large numbers of troops in most of the Warsaw Pact countries in Eastern Europe.

Hope it helped.