They tried to remove him sooner, but the conspiracies failed. It is also of interest to note that during his early reign he was a very popular and beloved figure.

According to Suetonius and Cassius Dio, Caligula did not start his reign as the insane man, as we know him today. Suetonius mentions that the crowds hailed him as "our star" the day he became emperor.

In the beginning of his reign, he was quite a popular figure among the senate and the people of Rome, mainly because they held his father -Germanicus Julius Caesar- in very high esteem. He started by doing some political and public reforms, such as abolished certain taxes, aided those who had lost property in fires and published accounts of public funds, which were well received by the people of Rome. He was close to Naevius Sutorius Macro, the prefect of the Praetorian guard, which even helped him ascent to the throne. Philo of Alexandria described him as the first emperor who was admired by everyone in "all the world, from the rising to the setting sun."

It was not until after an illness (if i remember correctly around 39AD), that changed his personality. As Suetonius puts it:

"So much for Caligula as emperor; we must now tell of his career as a monster."

Around that time Caligula also lost support from the senate over an issue that today is still unknown. Suetonius mentions also that there were several conspiracies that tried to assassinate him, but all failed.

About 3 years after his illness, he was assassinated. To me that does not seem like a long time. Nero, for example, is also considered an extravagant emperor and reigned for almost 14 years.

As Suetonius states, Caligula had his own personal guard of German soldiers completely separate from the Praetorian Guard, the traditional protectors of the emperor - so even though he lost the support of the Guard months before his assassination (likely due to Caligula forcing Macro, the prefect of the PG, to commit suicide), he could not be physically/directly challenged on the streets of Rome unless he was led into an ambush without this German guard there to protect him. Which he eventually was. In the immediate events after his death, these German soldiers kind of went on a rampage killing in sight. I don't believe they're mentioned again.