For my money, the best explanation of what happened is found in Michael Allen Williams still influential monograph Rethinking "Gnosticism": An Argument for Dismantling a Dubious Category (Princeton, 1996). In it he applied sociological theory to try and explain why certain versions of Christianity flourished and others did not. A paraphrased explanation would look something like this. Imagine you are a person in the third century Roman Empire. You are dissatisfied with Greco-Roman religious practices and are looking for something new. Three options present themselves:

1. A group of people really into Greek philosophy who think Plato really hit the nail on the head. They have also heard of this guy called Jesus and think he's kind of interesting.
2. A group of people who are into some guy called Jesus but have a strong community focus. They support each other materially and emotionally and you mostly just have to attend their meetings.
3. A group of people who are also into Jesus. To join them you have to give up on pretty much anything that brings you pleasure, be that sex, food, whatever.

Which group are you most likely to join? 1 could be interesting but it's not that different from the religious practices that you are already losing interest in. 3 seems pretty extreme so you would have to really believe in their interpretation. The most likely one for your average person is 2. It is different enough to what you already know to be distinctive while not being so extreme as to be off-putting.

This is Williams' argument in essence. The version of Christianity that won out won out for reasons that had very little to do with the apologetics of the early church leaders and a lot more to do with human psychology.

Of course there are other theories. With this topic there is fierce debate and one's theological views are highly influential. There are a lot of Christians who believe that the proto-orthodox won because they had the truth on their side (debateable) and point out that heretical groups often believed weirder ideas that can be demonstrably proven to be later developments. Those latter two points have some merit.

As far as the ancient evidence goes all we have is the apologetics and it's possible that the proto-orthodox factions won because they were more convincing. Linking in with Williams' argument though I think it is worth considering the fact that many of the heretical groups appear to have been rather thin on the ground when it comes to followers and also have lacked organisation when it came to establishing institutional structures.

Another highly influential work in this area is Walter Bauer's Orthodoxy and Heresy in Earliest Christianity (1934 in German, 1971 in English translation). One of his arguments was that the church in Rome emerged from persecution streamlined and with a high degree of motivation (otherwise it would not have survived). They put these skills to use in bringing other churches into its orbit, especially ones that were in more disarray when they suffered persecution. While many of his arguments have been discarded when it comes to the details, ideas like this still have some merit. In The Making of Orthodoxy: Essays in Honour of Henry Chadwick (Cambridge, 1989) Rowan Williams notes that many of the churches that emeged to become the orthodox church did seem to have strong links of communication with each other, seen in the letter writing that they took part in. Contrast this with the inumerable studies on Gnosticism over the past few decades. There is a lot of debate about the extent to which any gnostic groups can be said to have existed, and those that argue some did essentially admit that they were small groups, probably within existing church congregations, and when put under pressure by church leaders either went underground and died out, or were excommunicated and died off. A few useful sources here would include David Brakke, 'Self-differentiation among Christian groups: the Gnostics and their opponents' in The Cambridge History of Christianity , vol 1 (Cambridge, 2006); Mark Edwards, 'Christianity and Christianities' in T&T Clark Handbook of the Early Church (Bloomsbury, 2021); and Charles Freeman, A New History of Early Christianity (Yale, 2009).

Finally, to properly answer your question, it depends on what period you are talking about. The councils did not take place until Constantine initiated the first at Nicaea in 325 and at that point the proto-orthodox had imperial power on their side (although there were still swings between positions for centuries afterwards). This changed things somewhat, as did numbers. The library of Alexandria and the philosopher Hypatia were both killed by Christian mobs, but this was only something that could occur once Christians were in the majority. Prior to this, it does seem that approaches to establishing dominance combined rhetoric (apologetics in the main), alongside 'withdrawing communion' from those they disagreed with.