Good article on the subject: Erich Gruen - Augustus and the Making of the Principate in addition to the accounts of primary sources like Suetonius, Tacitus and Dio.

In short: Augustus was in a very precarious situation. He was very concerned with keeping up the appearance that Rome was still a republic. Few people had forgotten that Augustus' position initially was because of Caesar making Augustus his heir in his will. Augustus enjoyed exceptional power, but it was conferred on him personally, he did not occupy a position that contained all his power. The position of emperor did not yet exist. It could not be transferred to anyone when he died. He could not just name an heir either, because that would be seen as "acting kingly", and romans hated kings. Augustus publicly denied any sort of title or privilege that would be "too much". He denied being called dominus (master/lord) and other such titles. He was princeps, which means "the first one", roughly. He was also primus inter pares, which means "first among equals". He was not the emperor of Rome, but the first citized of Rome. One can argue that was semantics, at least in hindsight, but it was very important semantics. He collected prestige and authority, but he was no dictator or king. He made sure not to make much public and dramatic use of his power. He portrayed himself as a man of the people, not a man above the people, like the later emperors. Even still, he constantly had to try to evade accusations that he was a monarch with plans for a dynasty.

Still, Augustus had multiple close calls with death. He had some sort of stomach issue that made him very ill multiple times and bothered him, although he survived to his 70s. This made it clear that he had to make some sort of arrangements for his death. He knew that the chaos and civil war that followed the death of Caesar could easily be repeated. He had the power of multiple offices, divorced from the offices. This was a symbol of continuity and stability, but that would end with him unless he found some way to fix it. He could not designate a successor, but he could request a "colleague". And this is what he did. He took someone as a colleague, so that his extraordinary power would be associated with Augustus' extraordinary power. If he took a partner, and that partner made himself worthy of some of his own authority, there would be some continuity. So that when he died, it would be natural for his "partner" (in reality, his heir), continued on.

The problem that Augustus had was that the partners he took in died. Marcellus, his nephew, died of illness. Agrippa, his long time partner and general, and the man he though would succeed him, died of illness. The other man Augustus turned to, was Tiberius (his step-son and eventually son-in-law), who of course became the 2nd emperor. Tiberius was not the favorite of Augustus, he was known as a cold and misanthropic man, but extremely effective and talented, both as a military commander and as a civic administrator. He earned much of authority Augustus had and continuity seemed assured.

But at some point, the relationship between the two went bad. Tiberius withdrew from Rome and the public eye. There are many speculations as to why he did this, mostly to do with personal intrigue, but we will never know for sure. This threw a wrench in the plans of Augustus, since Tiberius seemingly had gone into retirement. He was forced to prematurely advance the careers of his teenage grandsons by Agrippa. Once his powers had run out, Tiberius sought permission to return to Rome as a private citizen as the grandsons, Gaius and Lucius, had come of age. Augustus denied his request, now that he had begun on the long road to make his grandsons deserve their powers, as Agrippa and Tiberius had done before them.

But misfortune once again struck Augustus. Lucius and Gaius died within two years of eachother due to illness. So he again, now in his sixties, had to return his focus to Tiberius as his only option. He adopted him as his son, together with the last living son of Agrippa, Agrippa, probably to consolidate the power and prestige of Agrippa and Augustus to Tiberius. Tiberius, a proven candidate, had no difficulty reclaiming his power. Agrippa, however, was banished. Tiberius and Augustus became a public duo. Whatever their feelings for eachother might have been, they appeared like father and son in public. Tiberius kept proving himself, as a military man abroad and as an administrator in Rome. So, when Augustus eventually died, Tiberius had almost as much power as Augustus. Continuity was preserved.

So, to answer your question: TL;DR: He was very innovative and smart in navigating the political minefield, so that he could have a successor, without formally having a successor. But ancient life expectancy made it so that the candidates died, and the one who didn't proved to be somewhat unreliable, even though he succeeded in the end.