I'd attribute this more to IR theory than to history. There is a history of hostilities between the United States & her allies and Russia, but the West also became allies with Germany & Japan swiftly after WWII.

It's usually accepted that there are seven Great Powers: the United States, United Kingdom, France, Germany, Japan, China, & Russia. The first five probably make up the closest alliance between Great Powers in world history. China & Russia tend to compete with the U.S. and her allies for influence in the area around China & Russia. Russia in Ukraine & Georgia, China in Korea & the South China Sea.

So, why does the United States compete with Russia & not Japan? The Japanese economy is three times the size of the Russia economy. It's not like Russia has an economic growth rate like China. Their power is waning and they most certainly aren't an emerging economy.

I might be tempted to attribute it to ideology, with some history mixed in. It's no coincidence that the United States, Britain, Germany, Japan, & France are liberal democratic states with free markets. Democracies tend to be relatively peaceful & stable.

I mentioned before the fact that Germany & Japan swiftly became American allies after the second World War. It's worth noting that these countries had relatively positive feelings about the American occupation and the United States helped set up pretty good governments. This created a generation whose opinions were very positive. This is shifting a bit with younger generations, but it is unlikely to affect Western foreign policy.

There has never been a time in history when Great Powers did not compete. The last 20 years is the closest we've been to that scenario in World History, mainly because of the relative power of the United States being greater than any Empire in World History, globalization providing an incentive for the Great Powers to keep other countries stable, and perhaps MAD.

Middle Ages to World War I

The hostility between Russia and the West pre-dates World War I.

I would say that it goes way back to the late middle ages when a lot of the Catholic west, especially the Holy Roman Empire, tried to repeatedly conquer and/or convert the slavs to the east. Most of the slavs were either pagan or Orthodox Christians. Some of the Popes wanted an active effort to convert the Orthodox and Pagans in these areas. To some degree, they were successful, and today much of Poland and Lithuania are Catholic nations, despite being slavs and in many parts, having previously been Orthodox Christians.

This automatically put the Rus, and later the Russians, at odds with the West. After the fall of the Roman Empire in the 15th Century and the fall of the Tatar Yoke, the Grand Duchy of Moscow believed that as the sole Orthodox nation free of any Muslim and Catholic oppressors, they were the successors to the Roman Empire and thus had the duty to protect all Orthodox peoples from their oppressors. You can see where this is going. Therefore, they felt a duty to protect, not just other northern Slavs, but also to protect Georgians, Greeks, Romanians and all southern Slavs.

This is one of the reasons its leader was called "Czar" or "Caesar". Their mindset was reinforced by the Ecumenical Patriarch in Constantinople, who was head of the churches in most of what is now Turkey, Greece, Romania, Ukraine, Russia and off-and-on, Serbia, Montenegro, Bulgaria, Bosnia-Herzegovina, Albania and Macedonia. Some of the Ecumenical Patriarchs directly referred to the Russians as the successors of the Roman Emperors, or the sole protectors of Orthodox Christians. This was reinforced when the Russian Church was granted complete independence of the Ecumenical Patriarchate in 1589, which is also when it was recognized as a fellow "Patriarch", which is the highest episcopal title in the Orthodox Church, recognizing a special authority and special place, as well as an apostolic foundation (the legend is that the Apostle Andrew traveled and was the first to preach to the Slavs).

Armed with this new authority, Russia treated it as its absolute duty to protect Orthodox Christians from both the Catholics in the West, and the Muslims in the South. There were a lot of wars beginning in this period that involved the Muscovites/Russians and Lithuania, the Ottoman Empire, Poland, Livonian Order, Swedish Empire, the Crimean Khanate, Great Britain, the Persian Empire, France, Spain etc...

All of this continued even into the 19th Century. In the 18th Century, Peter the Great was actually kind of controversial. The west loved Peter, because of how open he was to western ideas. But throughout much of Russia, he was actually hated because of his oppression of normal Russian culture and life. There are stories of him physically ripping the beards out of fellow Russians because he found them barbaric compared to groomed mustaches found commonly in some western cultures. He wasn't a big fan of the Russian Orthodox Church, and abolished the Patriarchate that had been established, instead forming a Holy Synod of Bishops so he could have Bishops appointed and control the Church easily (as one Bishop could easily say no, but in a synod he felt he could sway votes in his favor). This state of the Russian Church stayed the same into the 20th Century until the Soviets abolished the Holy Synod and allowed the Patriarchate to be re-established.

So Peter the Great, at least in Russia, remains a matter of controversy. Arguably, without him, Russia wouldn't have been able to continue to compete with the advance of western technology and military might. But with him, a lot of Russian culture was damaged, and had elements of western culture implanted onto it. Though the interesting part, is that because Russia adopted some of these things so late compared to the west, it attempted to make them uniquely Russian, and in some ways, surpassed its western counterparts. Which often happens when nations become late adopters of ideas or technologies, as they benefit from building from the ground up, rather than having to work within existing norms and ideas.

Despite having western ideas implanted into it, Russia was still at odds with the west. Its religious roots was still in Orthodox Christianity, and it still felt a strong duty to protect its brothers and sisters who were Orthodox. This included those who had previously converted to the Catholic Church and become "Byzantine Catholics", known as "Uniates" among Orthodox Christians.

When nations started to rebel against the Ottoman Empire in the 19th Century, Russia really took its sense of duty seriously. Repelling Napolean's invasion probably reinforced Russia's morale and reinforced their feeling that they could stand against westerners. So as the Ottoman Empire started to disintegrate from revolutions, Russia tried to take advantage. It aided once in the Greek Revolution in 1827, actually siding with the United Kingdom and France, soundly destroying the Ottoman fleet in the Battle of Navarino. In 1829, Russia once again defeated the Ottoman Empire, forcing them to sign a treaty with the Ottomans that allowed the Russians to demand, at their will, that the Ottomans had to cut off the Dardanelles to any and all ships (effectively restricting the Black Sea to the Russians & Ottomans).

In 1856, the Russians lost the Crimean War, which it fought against the Ottomans, French, British and Sardinians. This was the war that effectively destroyed Russian hopes of liberating Orthodox under the Ottomans and prevented Russia from gaining any territory from the dissolution from the Ottoman Empire. The French Empire were advocates of the Catholics under Ottoman rule, and allied with the Ottomans to contain the Russians, and advocate for the Catholics. Western involvement in this conflict was highly influenced by religion, as the Ottomans did support Russia's claim of being the protector of all Orthodox within the Ottoman Empire. Western nations like France really didn't like this, especially because a lot of holy sites that were previously in Orthodox hands, had fallen under Catholic control. During this period, the many Greeks (living in mainland Greece, and western Turkey) were still under the Ottomans, while many Armenians and Georgians also were still under Ottoman rule. As well as many Arab Orthodox in what is now Syria & Israel.

During this Crimean War, the Greeks tried to take advantage of it by rebelling once again against the Ottomans, hoping the Russians could aid them. But they were pressured (diplomatically and militarily) by the United Kingdom and France, to not get involved. The Greeks, therefore, didn't get involved on the side of their Russian brothers. When Russia lost the war, Greeks felt betrayed by western nations, losing at its chance once again to regain former Greek lands, including cities like Constantinople (which was still the city's name) and most of western Turkey. The Treaty of Paris, which was signed at the end of the conflict, completely crippled Russia's involvement in the region, as they couldn't sail their ships (other than trade) in the Black Sea anymore. They also were pressured to no longer involve themselves in the affairs of Orthodox under the Ottoman Empire.

This however, didn't prevent them from doing so. In the 1870s, the Bulgarians, Romanians, Serbians and Montenegrins (all Orthodox) finally rebelled against the Ottoman Empire, and Russia sided with them, helping in what limited way it could. The end of this war in 1878, led to the complete independence, or autonomy of the Romanians, Serbians, Montenegrins and Romanians from/within the Ottoman Empire.

Russia was unable to aid the Orthodox peoples of Bulgaria, Greece, Serbia Romania and Montenegro during the Balkans Wars. The First Balkan War ended with a victory, as the Serbians gained what is today Macedonia (originally a part of the Serbian Empire), the Bulgarians gained a lot of what is now Bulgaria. The Greeks gained more land in the north, as well as Crete, and the Great Powers were able to determine the fate of the Aegean Islands (most of which would later be given to Greece).

The second Balkan War actually involved Bulgaria fighting against its Orthodox allies, due to disputes in share of land after the Ottoman Empire was defeated in the first war. Greece ended up losing part of its territory to Albania, while Serbia gained more territory from Bulgaria.

World War I was sort of the swan song of Russian attempts to aid Orthodox in the Ottoman Empire, and Russia, as we know, ended up taking an early exit from World War I due to its own revolution.

Contrary to other comments, this is definitely a historical question; the problem is that some commenters have treated it as a political question with little historical relevance.

OPs question is pretty decently addressed in the Wikipedia article on The Great Game (the historic term for Anglo-Russian relations going back to the early 19th century. The TL;DR is basically as follows:

1. Russia's territory is big. Obviously. And the borders are so vast that generally speaking, the only security option is to expand to the sea in every direction, ESPECIALLY the Black Sea in order to have ports that won't freeze.

2. England placed huge value in her eastern empire. Specifically India. And her foreign policy hinged upon an abstract sense of 'keeping the road to India open'. Which meant a lot of meddling with Afghanistan and the Ottoman Empire.

See where this is going? Russia historically rallies her allies and Britain rallies her allies to butt heads over this theater for a century and long standing animosities develop.

In fact, many argue that the great game never really ended. The USSR effectively could've been seen to advance the same formula of Russian imperialism and American-Anglo foreign policy still heavily plays by the great games rules. And given what's going down in Crimea at this very moment, the ending of the Cold War did little to alter the landscape.

While not strictly a historical work, Robert Fisks memoirs, The Great War For Civilization, has this whole issue as a running theme. And it's an incredible book anyway. Other than that, I recall A Peace To End All Peace discussing The Great Game at length in the opening chapters.

"The West" is a very vague term that can mean many things. Even though the Russian Empire was far east geographically, it was still very much a Western power. As I see it, the Russian Federation is as Western as France or Germany, even if the Soviet era caused people to think in terms of Capitalist West vs. Communist East.

FYI this seems to be a politics question and 1994 is the cutoff for this sub.