What are the reasons causing such disparities? I guess for some cases, such abdications might be beneficial to the smooth transition of powers to their desired successors

Right on the money!

In Japan the system of emperors "abdicating," but still exercising power is referred to as Insei ("monastery administration") or the cloistered rule system. It really comes about because of the rather...complex court politics in the Heian-kyō (now Kyōto).

During the Heian Period, the Fujiwara clan exercised an enormous amount of social and political influence on the imperial court. From the 800s onwards, the Fujiwara had married into the imperial family and gotten themselves appointed to key administrative positions. Several Fujiwara family members eventually become regents. When the newly-throned emperors, was a child, a Fujiwara member acted as the Sesshō regent. Even when emperors reached adulthood, they still had the Fujiwara breathing down their necks - a Fujiwara clan member acted as the Kanpaku, a kind of advisor-cum-regent who really called the shots. Because of decades of intermarriage, virtually every early Heian emperor had a mothers or close relatives from the Fujiwara family. Several emperors tried, but largely failed, to limit Fujiwara influence - the Fujiwara were richer, better-connected, and more politically-experienced. The mother of virtually every emperor was a Fujiwara woman.

The Emperor Daigo enjoyed brief success during the 900s when he managed to sideline the Fujiwara and rule in his own right, without a regent. However, he failed to set up at robust succession arrangement. After he died, the Fujiwara regency system returned in even greater force. If an emperor crossed them, they would outmanuever and dethrone them.

In the late 1060s and early 1070s, the Emperor Go-Sanjō finally managed to buck the system. His mother wasn't from the Fujiwara family, which have him a degree of independence and distance precious emperors had lacked. Exploiting divisions within the Fujiwara family, he managed to issue several decrees that reduced the power of the regency. Determined to keep power, he did a rather unusual thing...he abdicated in favor of the new emperor, Shirakawa.

In 1086, Shirakawa also abdicated in favor of his four year old son, starting the Insei system of rule from retirement. Many emperors went off to monasteries and became a Daijō Hōō ("cloistered emperor"). Others lived in their own palaces in Heian-kyō. By abdicating, the retired emperors could get out of the Fujiwara-controlled court and set up their own system of administration with the In no chō ("Office of the Cloistered Emperor"). Although the Fujiwara still held high office (ex. Minister of the Center), the key political and administrative decisions were now in the hands of the retired emperors. The cloistered emperors even had their own army, the Hokumen no Bushi. It also gave the retired emperor the chance to effectively act as regents for their own sons, allowing them time to grow up and get established without becoming Fujiwara puppets or be forced out by rivals. Indeed, at multiple points the father and grandfather of the throned emperor were alive and "retired."

Shirakawa himself managed to outlast two of his successors to the throne, ruling from behind the scenes for 41 years until his death in 1129.

The insei system could lead to some tensions - the Hōgen Rebellion, for example, was the result of a succession crisis and the resulting conflict between the emperor and the retired emperor.