It has been on the rise in Japan, and the past decades in Western Scholarship as well, that the 公家 kuge, that is, the aristocrats in Kyōto which the Emperor was top, was not as powerless as often thought. Now I don't want to understate the importance of the Imperial bloodline and the cultural idea and icon that was transmitted for centuries, but just to give a new perspective, especially on new research.

First, the Kamakura Shogunate did not rule over the entire of Japan. Instead, it formed the more powerful half of a dyarchy, with the Shogunate ruling the east and the kuge ruling the west. There were a lot of disputes in which the Shogunate usually came out on top (for example, the Jōkyū War in 1221), but the Shogunate did not control the right to collect money and raise troops in Western Japan until the pressing need of the Mongol Invasions forced the kuge to hand over those rights, which ironically contributed greatly to the fall of the Kamakura Shogunate.

The fall of the Kamakura Shogunate gave rise to the Nanbokuchō, a drawn-out conflict of warrior clans who sided with the Emperor and warrior clans who sided with the Ashikaga, or Muromachi Shogunate. The conflict ended when the two sides reconciled - though still with the Shogunate as the more powerful partner - instead of conquest, in 1392. While the Ashikaga "ruled" all of Japan, it's "control" fluctuated and ironically was very weak in the East, the traditional stronghold of the warriors. And while the kuge lost its power in western Japan and its income was limited to personal estates, the Shogunate was not exactly in an all-ruling position. The Muromachi period is marked by the 守護 shugo trying to consolidate control over their domain, and the Shogunate trying to consolidate control over the shugo, the former being often much more successful than the latter. Rebellions against the Shogunate broke out in 1399, 1416, 1438, Shogun Ashikaga Yoshinori was assassinated and another rebellion launched in 1441, and finally the outbreak of the Kyōtoku War in 1454 that signaled the Shogunate's absolute loss of control in the Kantō. The Kyōtoku War continued into the Ōnin War (it in fact only ended years after the Ōnin War ended) and is as much a contribution to the start of the Sengoku as the latter. Given this then, I personally don't see the Shogunate as being in a position to do something as drastic as abolishing the Emperor.

The reconciliation process in 1392 while stripping the court of any formal control over Japan outside Kyōto, also gave it a place as the ritualistic head of the country and the source of legitimacy of warrior governments. With the chaos of the Sengoku, while the court was put in even harder times financially and had even less resources, it was very careful in doing everything it can to formalize it's role as the ritualistic head and source of legitimacy in both meticulous court rituals and equally meticulous record of formal ranks and titles, and their handing out. Despite no actual military power, they were so good at doing what they did with the power of rituals and appointments that, far from being powerless entities at the whim of warrior daimyos, the court was an oft-sought source of legitimacy for any clan and could and very much did use their position to negotiate often favorable settlements.

In the early 17th century, the Tokugawa Shogunate, in trying to consolidate power, sometimes with the court and sometimes against it, codified the activity of the court. Though stripped of rights to appoint high-ranking monks and bestow ranks, the court got a set of clear rules, responsibilities, and became the cultural head and trend-setters for all elites. The Shogunate drew clear lines marking who controlled what, and in doing so while stripping the court of some very important powers, also solidified its place. The new relationship is one which neither side needed to be much concerned with each other, and so perhaps became the more stereotypical Japanese emperor and court we have traditionally viewed. But I think it's important to recognize that the new Shogunate very much used and perhaps needed the Emperor's legitimacy and saw no point in trying to push the issue further. When the Shogunate was still very conscious of its hold on Japan and the precariousness of its position, it negotiated and got what it wanted from the kuge, in terms of important appointment powers, and gave the kuge what it wanted, a place as ritual head and cultural leader and ensured survival.

The 72nd Emperor of Japan, Shirakawa, invented the idea of Cloistered Emperors as a way to ensure succession. He passed the title of Emperor on to his son, Emperor Horikawa, and then continued to be the de facto ruler of Japan for 42 years until his death with the title Cloistered Emperor Shirakawa. Horikawa died in the mean time and Emperor Toba had both taken over as Emperor and abdicated by the time of Shirakawa's death.

Cloistered Emperor Toba, Shirakawa's grandson who had abdicated 6 years earlier, became the new de facto ruler of Japan after Shirakawa's death. Toba didn't like Shirakawa's Emperor appointment and forced his son and successor to the throne Sutoku to abdicate and replaced him with his, Toba's, 3 year old son Konoe. Sutoku ended up not having any power as a second Cloistered Emperor under Toba's rule.

Konoe died young at age 16 without leaving an heir. Sutoku wanted to put his own son on the throne but Toba denied him and put Go-Shirakawa, Sutoku's 28 year old brother, on the throne instead. When Toba died a year later civil war broke out between the 2 brothers and their allies in the form of the Hogen Rebellion. In order to win Go-Shirakawa promised power and prestige to the samurai, who helyp him put down the rebellion and exile his brother. However this left the Emperor weakened. No Cloistered Emperor was left to be the power behind the throne and the seat of the Emperor had been a puppet position for nearly 70 years. Go-Shirakawa quickly "retired" 2 years after the rebellion and became the ruling Cloistered Emperor, but he now had to compete for power with the daimyos.

A long power struggle ensued. Go-Shirakawa relied more and more on the samurai to maintain power. The next 2 emperors, sons of Go-Shirakawa, died young and without heirs. Emperor Takakura, fourth son of Go-Shirakawa, took the throne at age 7 and managed to last 12 years and produce several heirs. His son, Antoku, took the throne at age 2 and became a puppet of the Taira clan. 3 years later Antoku fled the capital rather than abdicate and Emperor Go-Toba, Takakura's fourth son, was put on the throne at age 3 with the backing of Cloistered Emperor Go-Shirakawa and the Minamoto clan, the samurai Go-Shirakawa had elevated. For the next 2 years there were 2 Emperors while the Taira clan and the Minamoto clan fought. Eventually the Minamoto clan cornered the young Emperor Antoku and a member of his household drowned him rather than allow him to fall into enemy hands. Cloistered Emperor Takakura has died by this point so the only Cloistered Emperor that exists at the end of all this conflict is Go-Shirakawa. Go-Shirakawa died when Go-Toba was12 years old and the shogunate was established to take on the role of power behind the throne. Go-Toba was forced to abdicate 6 years later and became a Cloistered Emperor, but his powers were greatly diminished and the position of Cloistered Emperor as de facto ruler of Japan was never reestablished, especially after Go-Toba's rebellion in an attempt to restore it was put down by the shogunate.

TL;DR The Emperor hadn't been the ruler of Japan for over a hundred years by the time the shogunate took over, they took power from the Cloistered Emperor. That system failed due to succession issues caused by Cloistered Emperor Toba and Cloistered Emperor Go-Shirakawa's inability to stabilize the government afterwards. When the shogunate took power they continued using the Emperor as the Cloistered Emperors had been, as a convenient figure head.

Edit: For clarity

The peak of emperor's power was probably be reached during the Asuka or Nara period - emperor had much to say in both fundamental and daily government issues, office grants and the like, starting from importing whole political system from China and forming new nobility class. From this point things went slowly but steadily downhill.

First, they started to lose power to court nobility - the end of Nara period is marked by moving the capital to today's Kyoto and speculated Emperor Kammu's motive was to break from aristocratic influence (by imposing costs on them and separating himself geographically). This didn't work out, by the middle of the Heian period Fujiwara regents had almost completely shut out emperors. After them came the warriors (samurai).

Both groups (the court and the samurai) used (and I'd risk saying needed) the throne for their claims of legitimacy. Nobility lived inside a bureaucratic system with codified administrative law and which depended on the emperors to grant offices and court ranks. They also didn't maintain personal military bands, so no real own executive power.

On the other hand, warriors generally had strong local influence coming from their personally sworn retainer bands, but national control was entirely different matter - the throne's splendor was needed for wider following. All shoguns were appointed by emperors. Muromachi shoguns used Go-Daigo as a pretext to overthrow of Kamakura bukufu) and received highest court rank and dabbed in aristocratic culture and customs. The great unifiers were no exception, Oda Nobunaga started with his march to Kyoto, Toyotomi Hideyoshi used court rank and his appointment as regent to counterweight his low birth status. Generally, imperial acclaim, even if forced, swelled own ranks and damped support for "rebels".

Where does emperors' immense symbolic value come from? Let's note that we have evidence of imperial line from around 500 AD, and semi-legendary even earlier. This means at least 600 years old for Fujiwara regents and 1100 years for Tokugawa Ieyasu. Religious aspect is also important - efforts to merge all Japanese local cults around the God-Emperors, descendants of Goddess Amaterasu started early (see 8th century's Kojiki). We can draw some parallels with papacy and Catholic Church in Europe. What is more, historic Japanese society put a lot of weight on precedent, tradition and custom at least outwardly.

Edo period wasn't much different. Shogunate maintained ceremonial relations with the court, as well as occasionally providing consorts for emperor or taking princesses as shoguns' wives. A short period of doubtful power came with arrival of Western ships - with Tokugawa weakened, some turned to Kyoto's opinion on foreign affairs. Ultimately, shogunate fell and new government put emperor on its banners - Meiji restoration.

Source: Cambridge History of Japan, vols. 1-4.