This is going to be a long answer, because it's intensely complicated.

First, some recommended reading: the GRU is markedly less well-studied than the KGB, because it has never courted the same degree of notoriety. The best text I can recommend is Victor Suvorov's Inside Soviet Military Intelligence. Suvorov is a former GRU officer who defected to Britain in the 1970s; it's not a scholarly history, but it's a fascinating insider account of the GRU's history, and includes an entire chapter on the complex relationship between KGB and GRU.

The KGB has a more substantive and varied bibliography, but my personal recommendation is the work of Christopher Andrew, a pioneering intelligence historian who has written three histories of the KGB (working with two major Soviet defectors), all of which touch on the complex relationship between the KGB and military intelligence:

• Christopher Andrew & Oleg Gordievsky, KGB: The Inside Story of its Foreign Operations from Lenin to Gorbachev (1990)
• Christopher Andrew & Vasili Mitrokhin, The Sword and the Shield: The Mitrokhin Archive and the Secret History of the KGB (2000)
• Christopher Andrew & Vasili Mitrokhin, The World Was Going Our Way: The KGB and the Battle for the Third World (2005)

Now, onto the actual answer to your question:

The rivalry was longstanding, and had a lot to do with which service held the prestige of being primarily responsible for foreign intelligence — but it was also part of a broader tension between the Communist Party and the Red Army and the internal ideological warfare that defined the early years of the USSR.

The KGB's lineage (from 1917 through 1954, it went through a series of names: Cheka, GPU, OGPU, NKVD, NKGB, MGB) is complex because foreign intelligence, the responsibility of the First Chief Directorate, was only one part of its work; it was also responsible for political policing, internal security and counterintelligence.

In fact, in its early years, foreign operations were relatively low priority for the Cheka. During the civil war and the early years of the Bolshevik state, it was far more concerned with combating internal counterrevolution (real or imagined) and consolidating Soviet power through terror. The Cheka's foreign department (INO) was created in 1920, and took several years to mature.

The GRU also finds its roots in the civil war, and the intelligence staff established by Trotsky during his tenure as People's Commissar for Military and Naval Affairs — and the rivalry between Chekists and military intelligence began there too:

From its inception, military intelligence suffered the greatest possible antagonism from the Tchekists. The Tcheka had its own central agent network and an agent network in local areas. The Tchekists jealously guarded their right to have secret agents and could not resign themselves to the idea that anyone else was operating similar secret networks.

— Victor Suvorov, Inside Soviet Military Intelligence (1984)

In those early years, the Cheka made a point of infiltrating military intelligence, and there were periodic purges of the nascent GRU. Following a tense confrontation between the Party and the Army at the Eighth Party Congress in the summer of 1919, safeguards were put in place to ensure political control of the army — and its intelligence service. The head of the GRU was almost invariably a Cheka (or Cheka-approved) appointee, but it would remain a military intelligence service, albeit one occupying a somewhat uncomfortable position between an army general staff that mistrusted it and a state security organ that hated it.

But despite a troubled birth, the GRU was much quicker to establish overseas intelligence-gathering capabilities than the Cheka:

The agent network of the GRU was reinforced at almost lightning speed. There are several reasons for this. Firstly, inside Russia after the Revolution, in her central provinces alone, there were more than four million foreigners: Germans, Austrians, Hungarians, Poles, Slovaks, Czechs, Koreans, Bulgars, Serbs, Croats and others. Most of them were former prisoners of war. More than three hundred thousand of them voluntarily enlisted in the Red Army. There was no need to recruit such people. The overwhelming majority of them were convinced, fanatical communists. Military intelligence simply sent them off to their own countries as GRU agents.

— Suvorov.

Throughout the 1920s, the GRU and the GPU/OGPU jostled for primacy in the foreign intelligence realm. Both became pawns in the power games of the Stalin era, and both were impacted by the sweeping purges of that time. The politicking reached a bizarre conclusion in 1947 when Stalin ordered the merger of the foreign intelligence departments of the MGB and the GRU into a new, unified Committee of Information (KI).

The merger was a disaster; it had been proposed partly as a response to the establishment of the CIA and an effort to defragment Soviet intelligence — but it was also an attempt to dilute Lavrentiy Beria's power-base by placing foreign intelligence under the purview of the loyal Stalinist foreign minister Vyacheslav Molotov. In the event, it created a divided and deeply ineffectual agency, and inter-agency politics began unravelling the merger almost immediately:

In the summer of 1948, after a prolonged dispute with Molotov, Marshal Nikolai Aleksandrovich Bulganin, Minister for the Armed Forces, began withdrawing military intelligence personnel from KI control and returning them to the GRU. Probably with the support of Beria, [Minister for State Security Vladimir] Abukamov then embarked on a long drawn out struggle to recover control of the remnants of the KI. At the end of 1948 all residency officers in the EM (Russian émigré) and SK (Soviet colonies abroad) Lines returned to the MGB. The KI was finally wound up and the rest of its foreign intelligence responsibilities returned to the MGB late in 1951.

— Christopher Andrew & Vasili Mitrokhin, The Sword and the Shield (2000)

The essential difference between the KGB and the GRU in the post-Stalin era was a somewhat uncomfortable division of responsibilities: the GRU has always been strongly focused on military and scientific/technological intelligence; the KGB was far more interested in political intelligence. The problem is that intelligence material and the sources who provide it frequently defy simple categorisation.

The GRU eventually grew to be substantially bigger (in terms of manpower, budget and agent network size) than the First Chief Directorate, and clawed back much of its operational and political independence. But its evolution, and the re-definition of the power dynamic between the two took many years.

The two agencies found themselves treading on each other's toes frequently in their operations abroad — and the GRU would often lose these turf wars: in the early 1940s, for example, the NKVD aggressively and successfully seized control of the atom spy network targeting the US-UK nuclear weapons programme, taking over the running of key agents like Klaus Fuchs and Melita Norwood who had previously been cultivated and controlled by the GRU. Even more problematically for the overall integrity of Soviet intelligence, responsibility for signals intelligence was divided between the two: the GRU operated a massive military SIGINT programme, targeting military communications; the KGB's Eighth Chief Directorate operated a similarly expansive effort against diplomatic and political traffic.

There were periods of collaboration, most notably the 1980s RYAN operation to collect intelligence on a presumed American plan for a nuclear first strike against the USSR. But the two organisations never trusted one another, and the relationship was never really anything but adversarial. At the beginning, it was an existential battle: the Cheka sought to co-opt and control military intelligence; later, it was more a rivalry of tradecraft.