So I have written an asnwer on a similar topic to this, though it was outside of the time frame that you're looking at specifically, it focuses on the "Gothic" invasions of the Empire in Late Antiquity, after the heyday of the Roman Empire under Trajan and other 2nd century emperors. I will repost it below though for your perusal, and edit it a little bit to be more pertinent to the question that you asked.

---

So first off, who were the Goths, how did they end up in Roman land, what was their relationship with the Empire? According to Jordanes, the Goths traced their ancestry from Scandinavia and had settled around the Black Sea for some time, ending up on the Roman radar in the middle of the 3rd Century Crisis. They had led prosperous lives on the fringes of the Romans world until the depredations of the Huns forced them to flee their ravages and try to escape to the safety of Roman territory. The Goths split into Visigoths and Ostrogoths at this time, the Ostrogoths remaining under Hunnic authority for a time while the Visigoths crossed into Roman territory.

Traditionally, the Goths are marked as the epitome of the participants in the Voelkewanderung. An entire society uprooted by the onslaught of the Steppe peoples, who then poured over the Roman borders in their thousands bringing their entire populations in search for a new home. The Romans let in some of the refugees but not all, eventually enmities between the groups flaired up and the Goths were responsible for some of the worst disasters in Roman history, the battle of Adrianople and the 410 sack of Rome come to mind. Eventually the Visigoths settled in Aquitaine and Iberia and the Ostrogoths took over Italy. They were the latest, and most successful, in a line of Germanic interlopers into Mediterranean society that stretched back to the earliest days of Rome's existence, the tribes and peoples differed but their opposition to Rome remained from the Teutones, to the Alemanni, to the Marcomanni, and finally culminated with the Goths.

From this traditional view we get a lot of the misconceptions that come down to the modern day.

This view has a LOT of flaws in it.

Any attempts to reconstruct the migratory past of a pre-literate society are always going to run into some large issues. There is archaeological evidence of a (partly) Germanic culture occupying space around the Black Sea that seems to have fallen apart alongside the rise of the Huns, the Chernyakhov culture. However any attempt to solidly connect this culture to the Goths is going to be tenuous. Efforts to construct a migratory history of the Goths based almost solely off of Roman accounts, which were often written well outside any realm of contemporary, are doomed to failure.

However the biggest issue with this traditional view, and this gets to the gist of your question, is that the Goths were perfectly willing to not only work alongside but fight for the Romans! And this was true of all of the "Germanic" and "barbarian" peoples of the Roman frontier. They were not a separate and distinct entity from the Romans in all cases. They had extensive trade relations, served in the Roman army, lived at times inside Roman borders. They did not arrive on the Danube river looking for Roman blood and maraud the countryside until they got their fill of it. Indeed it is hotly debated just when (if ever) Gothic identity solidified around "visigoths" and "ostrogoths". Peter Heather argues that this process did not happen until the various groups that all comprised the forces under Alaric were in Roman territory. The Germanic peoples were familiar with the Romans, and even the differences in material culture, such as the availability of armor, were not the difference makers that we often assume them to be. Armor is undoubtedly effective and useful, but it is not invulnerable, and Germanic groups from Arminius down to Alaric found ways to either neutralize Roman advantages in cohesion and armament with tactics, numerical superiority, and eventually technological and armament parity. The differences between a Roman legionnaire in the 4th-5th centuries and his Germanic warrior opponent were likely not many.

Throughout their interactions with the Romans there were Gothic forces who clashed with the Romans then turned around and fought alongside them, only to then end up back at war with the Romans. The "visigoths" fit this to a T.

They first arrived, fleeing the Huns supposedly, in the late 300's and strike a deal with Emperor Valens for safe haven. They eventually rebelled due to poor treatment by the Romans and delivered several defeats to the Romans. Peace was eventually made and the Goths moved on through the Empire. They were then roused to violence once again after the Roman general Sticlho was executed and the Goths were targeted by the Romans. This is what prompted Alaric's attack on Rome. Indeed following Alaric's sack of Rome the Visigoths established themselves in southern Gaul as foederati of the Romans. They participated in Roman wars against the other "migratory" groups at this time such as the Suebi, Huns, and Alans. Eventually this relationship broke down and the Visigoths carved out their own realm in southern Gaul and Iberia.

Now this timeline of events is a holdover of older scholarship, however there is little basis to overturn the basic order of these events. The big controversies largely stem from disputes over the number, ethnic/cultural make up, and composition (all military, civilians mixed in, mostly civilian, etc) of these groups. This was not a question of the Romans being unable or unwilling to deal with the Goths and other invaders on their territory. In fact they dealt with them as both allies and enemies quite extensively. Relations ranged from subservience, outright warfare, to alliance, and eventually the Ostrogoths preserved a host of Roman institutions in Italy. The gulf between the Romans and their opponents was not as clear cut as mo0dern media would have you believe. The Romans were not all armor clad professional soldiers who cut a clear and distinct picture against the less advanced Germanic peoples. Due to the centuries of interaction between the two, the Romans and Germanic peoples often fought in similar ways with similar armament.

Now, why didn't the Romans deal with them once and for all during one of the periods of warfare between the two groups?

Rome at this time, the alte 3rd-5th centuries was not the Rome of centuries prior. The Goths had achieved quite successful results against the Romans in the past as well. The infamous battle of Adrianople when the Goths managed to destroy an entire Roman army and kill the Emperor Valens comes to mind. In addition to this, the Goths were only one off and on enemy that the Romans had to deal with. The other groups, that the Goths helped fight as often as not, also engaged the attention of the Romans, not to mention the rival Sassanids, Civil Wars, and Imperial pretenders that so often plagued the late Empire.

---

Sources

I've drawn quite heavily from several of Peter Heather's works. The Fall of the Roman Empire, Empires and Barbarians, and The Goths

Michael Kulikowski's,* Rome's Gothic Wars: From the Third Century to Alaric*

There are also a host of ancient authors who all deal with the Goths, but the vary in terms of reliability from "generally somewhat trustworthy to get basic chronology" to "Ancient equivalent of Ancient Aliens", I'm exaggerating of course, but it is best to take anything that they write that does not deal directly with Roman/Goth interactions relatively recently after they occurred with enough salt to season a whole McDonald's worth of fries. I've put a few of them down below.

Jordanes, he also used an earlier, and now lost account, written by Cassiodorus, to write his Getica

The Augustan History written by an anonymous author

Ammianus Marcellinus's, Historiae