NB: I will refer to the "twelve disciples" in this post, as "Apostles" is a bit ambiguous and a can also refer to e.g. Paul, the "Apostle to the Gentiles".

"Do scholars agree..." is obviously a bit of a tricky question when it comes to biblical criticism - scholars agree that Jesus existed and a few details about his life (He was a Galilean Jew, baptised by John, executed by Pontius Pilate) but not very much else. But I know the views of Bart Ehrman on this question well as well as the arguments he makes in terms of primary sources, as well as the few counterarguments around.

In the sense that there were twelve particular disciples with particular names who did particular things, no, there is no agreement on this, and there is good reason for that (viz., the gospels and Acts do not agree on the names of these twelve particular disciples or the things they did). In the sense that that Jesus had a group of followers known as "the Twelve", who may or may not have numbered exactly twelve at various points in time, I know at least Bart Ehrman argues that this was the case, based on a couple of different features of the Gospels.

Perhaps the most obvious one is the featuring of Judas Iscariot - it would have been very obvious to the reader that they were not "twelve" any more after his death. And so, it would not make very much sense to invent such a group for symbolic reasons. As a matter of fact, in Acts 1:12-26, this problem is remedied by drawing lots to "replace" Judas. Acts is widely viewed as being very questionable as a historical source (i.e. of a different genre than the Gospels) - so this is exactly the place you'd look for convenient "fixes" to uncomfortable aspects of the Gospels.

The problem with the group goes further. One of the most fascinating passages in the Gospels is Matt 19:28:

Jesus said to [the disciples], "At the renewal of all things, when the Son of Man sits on his glorious throne, you who have followed me will also sit on twelve thrones, judging the twelve tribes of Israel, believe me."

This is an orphan passage that does not appear anywhere else in the gospels, and one can easily see why - Jesus prophecies that Judas will be a prince in Israel! (As an aside - Bart Ehrman makes a fascinating argument based on this passage and others: Jesus being the master of the twelve, would presumably have been the great king of Israel in the proposed scenario. This is exactly what Pilate (but not the Sanhendrin) accuses Jesus of being. Judas revealing Jesus' location does not make much sense, since the Sanhendrin could simply have followed him. Thus, Ehrman argues, it is plausible that Judas actually went to the Roman authorities to reveal Jesus' plans to make himself king of Israel.) The fact that the passage sticks out like a sore thumb and has no easy explanation in terms of narrative devices makes for a fairly compelling argument that it is indeed an early saying attributed to Jesus.

One argument made against the existence of a "Twelve" is that they were a literary device to match the twelve tribes of Israel. Of course, that would equally well be a basis for a first-century Jew with prophetic or even messianic pretentions to assemble a group by this name, which the above-cited passage supports. And the books of the NT really look more like they are trying to fit twelve people into a prior agreed-upon set (where only a few, like Simon Peter, would have been universally agreed upon)

To wrap it up, it is interesting to note that association with one of the disciples was an important aspect of early Christianity that survives today - the notion of apostolic succession. For a church to have authority, it needed some claim to trace its founding back to one of Jesus' close followers. To constrain oneself to twelve disciples for literary reasons would seem to hinder, rather than expediate the making of such claims.

For sources and further reading - I am having trouble keeping apart all of Ehrman's works I have read in my head, but Jesus, Apocalyptic Prophet of the new Milennium would probably be your most accessible bet. Otherwise, you can't go wrong with The New Testament: A Historical Introduction to Early Christian Writings. I especially like Ehrman's works because he is good at summarizing arguments made in his field, even if he does not agree with them, and to differentiate between when he adheres to "mainstream" views within his field, and where he has more idiosyncratic interpretations. He also has a very clear and plain writing style that makes his arguments very easy to follow and understand.