First of all, /u/HatMaster12's remarks on whether the Republican state was "destined" to collapse is entirely correct. Syme theorized that the Republican state as it existed in the 1st Century was inadequate with respect to the realities of Roman power as it existed then, as opposed to the realities that had been at the time of the state's establishment, but while Syme thought that the most logical end result of the instability of the Republican state by the end of its lifetime was the seizure of power by an individual he did not discount the possibility that other solutions may have presented themselves. To Syme the dissolution of the Republican state was not a fault of the system itself, which was quite resilient, but was the fault of the nobiles who, echoing Cicero's disgust with Hortensius and the other piscinarii ("fish-fanciers," guys like Hortensius who, Cicero remarks with revulsion, were more interested in attaining consular rank and hanging out in their villas with their fish ponds than worrying about how the state was actually run), sat on their hands and were more concerned with bickering among themselves and accumulating minor individual honors than actually running the state or resolving crises in a logical way. To Syme this was a great tragedy, that the people who had initially made the Republican state great basically devoured themselves. But it needn't have turned out that way. Badian once remarked that the Republican state was nothing if not resilient and capable of change--it had reformed itself drastically several times already in response to crises and was capable of doing so time and time again. There was no failing inherent in the Republican state, the Republican state collapsed due to a number of factors that the state failed to respond to properly, not because it was somehow set up to fail. It's far from impossible to imagine a scenario in which the Republican state worked out a way to more systematically adapt the institutions of the state to the provinces--they had already figured it out in Italy after the Social War, and the proconsular system of provincial government, which everybody acknowledges was weak, was an attempt to improve on the older, rather ad hoc systems of provincial administration that had existed immediately after the Punic Wars. It's pretty easy to imagine the proconsular system of provincial administration being further refined and adapted as the Republican state evolved. And it's not hard to imagine many of the crises going the other way. Certainly Caesar's civil war, which was easily the most important crisis of the 1st Century, could've been very easily avoided had the Catonians been willing to negotiate with Caesar rather than obsessively insisting on prosecuting him, even breaking the law to do so (Cicero in one of his letters comments that he thinks that Cato's obsession with bringing Caesar to court is more damaging to the state than helpful and he wishes Cato would stop and think for a moment)

Why didn't Rome form a new Republic (say via radical constitutional change) or even collapse under its own weight?

Well...it sort of did. To call the Principate a monarchy is a bit too simplistic. True, the princeps had sweeping individual power, roughly equivalent to that of many monarchs, but there's a reason we call it the Principate (and then later the Dominate) rather than the monarchy. And it's not just because the princeps didn't call himself a king, there were fundamental differences in the way the Principate worked. Under Augustus free elections continued, magistrates continued to propose legislation, the various trappings of the Republican state continued to exist. The only difference was that the princeps watched over everything and made sure it was running properly, preventing disputes and crises among the various magistrates and so forth. Unlike Caesar, who had been established in a pre-existing magistracy (although technically the dictatorship had lapsed his was a revival of the practice), the princeps was not a magistracy. The princeps was a private citizen, he held no elected office (and Augustus made a big deal of renouncing the consulship), but held censorial and tribunician powers along with the imperium maius. He was thus a private individual with great power who existed outside of the state. So in a way the Republican state did continue to exist, although the princeps became understood as essentially a hereditary magistracy more and more as time went on, culminating with the Flavians and especially Domitian, whose title of dominus et deus actually persisted past his death. But in actual legal practice the Republican state continued to exist, just with a sort of overseer, and Augustus spoke of having re-established the Republic (rem publicam ex mea potestate in senatus populique Romani arbitrium transtuli, as he puts it himself in the Res Gestae). Not the same as a monarchy, and indeed the most important direct power of the princeps was not legislative or anything like that, it was the imperium maius. The imperium maius granted supreme military command the the princeps, which entailed the regulation of the recruitment of legions (during the late Republic every promagisterial governor recruited at least four legions, resulting in a lot of armies running around) and the provincial authority over the so-called "Imperial" provinces, those that the princeps governed directly. What that really meant in practice is that the princeps appointed provincial governors in his provinces (mostly the frontier) directly. But provincial command was still granted to praetorians and consulars in the other provinces, just like it had been before. So even then the division between where the powers of the princeps began and ended is tricky. In practice the princeps generally commanded by great influence and auctoritas rather than using any legal power (not that he lacked enormous legal powers), at least up until the Flavians. The enormous personal fortune of the emperor and his power of imperium maius was generally the basis for his influence, rather than any legal powers, so it's not really right to call it a monarchy.

I'd also add that when /u/HatMaster12 says that the Roman state had no written constitution that's maybe even putting it in our modern terms too much. It's not really that the Roman state had no written constitution, it's more like that they had no constitution. Now "constitutional theory" as a concept can be debated over, but the Roman state did not have a body of principles that we would immediately recognize as a constitution, though it may well qualify in terms of political theory. The "Roman constitution" was simply the body of laws that existed in their entirety. Roman law didn't generally discard or amend laws, the way modern constitutions and things like the Supreme Court do. When a law was outdated or disputed it was either ignored or written over. But that law still technically existed unless specifically contradicted by more recent one, which could cause problems and confusion (for example, during Caesar's consulship Bibulus invoked an extremely ancient religious law to stop elections which hadn't been used in centuries, which Caesar simply ignored since it was no longer relevant--on the basis of this there was some legal squabbling, since his enemies wanted to make his legislation invalid by this law). "Radical constitutional change" wouldn't really make sense to a Roman--the closest thing that we have to a constitution is probably the way Cicero understands the mos maiorum, which is that it's not just the actual laws of our fathers, but the principles and morals on which they were established and according to which our fathers lived and ran the state. That's rather a hazy concept. But in a way the Principate was a radical constitutional change maintaining the existence of the Republican state, for the reasons that I've given. The princeps was removed from the regular workings of the state--he had power, but he was not a part of the normal business of the state, which continued more or less as it had in the past. The degree to which the princeps' power was more-or-less absolute is debatable (some people think it wasn't really as absolute as Syme thought, and though I personally tend to favor Syme's thinking on the princeps there's a great deal of merit to this idea) but the fact is that he was removed from the state and in all other ways the state continued to function as it normally did. Until later in the Principate powers like the imperium maius, tribunician power, and censorial power all had to be granted explicitly by law to the individual of the princeps. It's unclear whether the lex de imperio Vespasiani was the first time that the powers of the princeps had been written into a single law all at once (which I find more likely, since we know that the Julio-Claudians had, for example, their tribunician power granted by particular laws, and that sometimes it even had to be renewed) or whether previous emperors had had this sort of all-encompassing law attached to them, but the law is an indication that as late as the Flavians the actual powers of the princeps legally were not bound to the title itself, they were granted legally to the individual by regular electoral process. No matter how coerced this may have been (and Vespasian's legions certainly helped smooth things out) the Republican legal process still existed and was vital for the way the princeps' powers functioned. Constitutionally, then, is the Principate a radical reform of the Roman state, or is it a complete dismantling of the Republican state, to be replaced by something new? Or is it both? The line starts to get blurred there, legally speaking.

I would strongly contest the idea that the Republic was “destined” to fail. Historians generally refrain from describing historical processes as preordained, as they ignore the influence of broader trends and individual agency. More simply, it is a fallacy to say that a historical outcome was the natural course of events, as such a belief can only be arrived at through hindsight.

If anything, the history of the early Republic demonstrates how dynamic it’s governmental institutions were (remember Rome had no written constitution, constitutions as we conceptualize them are modern inventions), as Rome was able to exert political control over most of modern Italy, no small achievement for an originally small Latin city. That even Hannibal was unable to detach a significant number of allied Italian cities from Rome is a testament to the political apparatus it established. The annual nature of consular (and proconsular) commands provided Rome with a pool of experienced commanders, both to command armies in the field and to shape strategy (as best as communication allowed) from an informed position in the Senate. Rome’s dominion over Italy provided access to Italian manpower, which provided a critical advantage in wars against the various Hellenistic states, several of which (such as the Seleucids and Ptolemaic Egypt) relied upon an increasingly dwindling pool of Greco-Macedonian colonists for recruits.

Yet even with these successes, it must be remembered that Rome remained a polis society. It’s institutions were designed to govern a city and it’s hinterland. While they worked well enough within the legal framework (comprised of differing types of alliance agreements, to put it simply) Rome established in Italy, they became stressed as Rome acquired overseas territories. /u/XenophonTheAthenian knows far more about this than I do, and can hopefully provide a more comprehensive answer (as well as address your question on Augustus). Hopefully, my short summary demonstrates that the government of the Republic was not “destined to fail”, as it proved quite effective during it’s first few centuries. The institutional crises suffered during the second and first centuries B.C. result not from a latent “design flaw”, but rather because of fundamental changes in Roman politics and in Roman society.