You're asking an "ought" question, so it's important to know what the criteria are for that "ought" (or whether "religion aside" is even possible). A lot of considerations functioned in the Church's decisions about the canon through successive assemblies. However, we can say at minimum that the Church expected the texts it selected to accurately reflect apostolic teaching, which it deemed inspired and authoritative.

This is something the historian may be in a position to say something about. However, the subject is controversial, and experts are far from unanimous. And even this criterion, which dates to the 2nd century, gets muddy.

For example, what counted as "apostolic teaching"? Many of the Gnostic schools claimed access to "secret" apostolic testimonies. The 2nd-century heretic Marcion believed only Paul faithfully conveyed the gospel of Christ; the others went astray. Hence Marcion's canon included only a purged version of Luke's gospel plus ten "authentic" Pauline epistles: Galatians, 1 and 2 Corinthians, Romans, 1 and 2 Thessalonians, Ephesians, Colossians, Philippians, and Philemon. However, by "apostolic teaching," most of the Church Fathers meant teaching in line with the "rule of faith" delivered in oral and written form by the original apostles, six in particular: Matthew, Peter, Paul, John, James, and Jude. And unlike Marcion, most were not in a hurry to figure out exactly which books counted or did not, which is why our complete list of canonical NT books doesn't appear until a letter of Athanasius in 367. For centuries after Christ, different communities used slightly different sets of books as scripture (that is, for reading during services). The term "canon" itself did not appear until the fourth century.

Despite that fuzziness, some broad statements can be made. The synoptic gospels and the major Pauline epistles were universally accepted "except among the more extreme proponents of Jewish Christianity" (Ackroyd and Evans). These formed the "core" of the early Church's scripture, and even fringe groups such as the gnostics made use of them. The rest of the canonical NT was used some places, but was unknown or not recognized in others until a certain point.

What do today's historians say about these texts? Well, every one of them has been questioned, and textual criticism suggests a larger group of writers were involved in their creation than tradition has passed down. The four gospels, especially Mark and Matthew, are very old, and probably do represent original apostolic teachings. Luke and Acts were less privileged until the second century, when they became very important for a heresy-embattled Church. Paul probably wrote the epistles attributed to him, except for the Pastorals (Timothy and Titus), with Ephesians, Colossians, and 2 Thessalonians being divisive among scholars. Who wrote these, if not Paul? Colossians, Thessalonians, and Ephesians were written, if not by Paul, by devoted followers. Who wrote the Pastoral Epistles is an open question.

What about the other epistles in the New Testament, which it appears some early Christian communities used and others did not? 1 Peter was widely received early on as being authentically Petrine. 2 Peter and Jude were widely accepted by the end of the 2nd century. Hebrews was attributed to Paul by some in the early Church, but nobody accepts that now; nevertheless, it holds an important place in the canon. The origin of James is highly controversial, with some scholars claiming it was not even authored by a Christian; it was one of the last books to receive recognition from the Church.

However, in all this I have left out the "Johannine" writings: the gospel, epistles, and apocalypse all attributed to St John. These books and their relationship to one another are disputed, but many would suggest they are products of a "school" that has its roots in their reputed author, John the Apostle. The Gospel of John and 1 John were accepted earliest; at the other end, Revelation was one of a number of Christian apocalypses circulating in the early Church, distinguishing itself perhaps most by its compelling literary construction and compatibility with the rest of the New Testament.

In short, regarding the canonical texts of the NT, there's a spectrum of plausibility as to which authentically stem from the hand or teaching of a recognized Apostle, and few clear answers. I don't have time at the moment to say anything about other books that claimed scriptural status in the early church; I'll come back to that later, or leave it to someone else.

Sources

Bruce Metzger, The Canon of the New Testament

P. R. Ackroyd and C. F. Evans, The Cambridge History of the Bible