So this is both a really good but also extremely complex question which I wish I could give a definitive answer to but know that I am both relatively understudied (I only have a Bachelor's in Comparative Literature, I studied Baroque Spanish Literature) and contextual (I am a student of literature, not a student of the history of literature).

Two things we must break down:

1. Modernism. Do we mean literary modernism? I think so given we're talking about Don Quixote but feel free to correct me here, I just wonder the context in which you hear people call it the "first modern novel" as that might help specify what we're talking about when we say "modern novel."
2. The novel. The first novel is a contested area of study (some say The Golden Ass others disagree). For now, I am going to assume that novel means a published work that could be widely read. I am choosing this definition only because of the context of Don Quixote's publication after novels such as Tirant lo Blanch and alongside the massive amount of plays by writers like Lope de Vega.

While literary modernism is considered relatively new and delimited in time see Ezra Pound's often quoted line "Make it New" (which I am struggling to find a source for). Peter Childs's writes in his book Modernism:

Modernist art is, in most critical usage, reckoned to be the art of what Harold Rosenburg calls "the tradition of the new". It is experimental, formally complex, elliptical, contains elements of decreation as well as creation and tends to associate notions of the artist's freedom from realism, materialism, traditional genre and form, with notions of cultural apocalypse and disaster.^1

In poetry the modernist turn associated with Pound is often focused on a heightened awareness of word choice, "directly treating the thing," and musical composition.^2

So both Ezra Pound's writings and the characteristics of modernism that Harold Rosenburg identifies place it sometime after the 1890s, the earliest, or after the 1930s. As Childs writes "We can regard it as a timebound concept (say 1890 to 1930) or a timeless one (including Sterne, Donne, Villon, Ronsard). The best focus remains a body of major writers (James, Conrad, Proust, Mann, Gide, Kafka, Svevo, Joyce, Musil, Faulkner in fiction."^1 Childs also cites Pound's famous phrase to say that "Modernist writers therefore struggled, in Ezra Pound's brief phrase, to 'make it new,' to modify if not overturn existing modes of representation" in their attempts "to render human subjectivity in ways more real than [literary] realism: to represent consciousness, perception, emotion, meaning and the individual's relation to society."^1

But all of this is contextual information. None of which directly says "why is Don Quixote considered the first modern novel"

To answer this question, I wish to turn towards Maurice Blanchot. In Blanchot's essay "Literature and the Right to Death" he writes what perhaps comes to define the field of literature, at least in my opinion, to this day:

Let us suppose that literature begins at the moment when literature becomes a question. This question is not the same as a writer's doubts or scruples. If he happens to ask himself questions as he writes, that is his concern; if he is absorbed by what he is writing and indifferent to the possibility of writing it, if he is not even thinking about anything, that is his right and his good luck. But one thing is still true: as soon as the page has been written, the question which kept interrogating the writer while he was writing-though he may not have been aware of it-is now present on the page; and now the same question lies silent within the work, waiting for a reader to approach-any kind of reader, shallow or profound; this question is addressed to language, behind the person who is writing and the person who is reading, by language which has become literature. (pp. 300-301)^3

Blanchot's focus here on a question, perhaps unconscious, to the writer and reader of a written work is what defines literature. Literature is an action of language interrogating itself through the history of reading and writing (since every writer is a reader and vice versa according to Borges's works Pierre Menard, Author of the Quixote and Kafka and His Precursors).

As such the key aspects of Don Quixote as a work which are often pointed to as aspects of literary modernism are:

1. Harold Bloom's observation in the introduction to Edith Grossman's translation of Don Quixote that "It is true that I cannot think of any other work in which the relations between words and deeds are as ambiguous as in Don Quixote, except (once again) for Hamlet" (p. xxviii). This ambiguity comes to reflect both the focus on the qualities of words and subjectivity proposed by Pound and on the "question" of Blanchot.

2. Childs cites Don Quixote's parody of "the previously dominant mode of prose writing, the Romance"^1

3. Bloom also cites Kafka's reading of Don Quixote on p. xxxiv of his introduction to the book.

4. Childs identifies "elements of religious skepticism, deep introspection, technical and formal experimentation, cerebral gameplaying, linguistic innovation, self-referentiality, misanthropic despair overlaid with humor, philosophical speculation, loss of faith and cultural exhaustion" as all exemplifying "the preoccupations of Modernist writing."^1 Many of which can be found in the pages of Don Quixote. Specifically I would highlight the poems at the beginning of the book which end with Babieca and Rocinante and the prologues to the first and second parts of Don Quixote---the first prologue acting as a way of distancing Cervantes from being the author of the book "But though I seem to be the father, I am the stepfather of Don Quixote" and the second prologue addressing the "illustrious or perhaps plebeian reader" (pp. 3 & 454).

The answer I would provide is that these pre-modern novels often lacked this sort of self-referentiality, formal experimentation, and linguistic innovation in addition to Don Quixote being read by and as contemporary with authors within literary modernism by the readers and writers answering the question behind Cervantes's pages.

I do hope someone more qualified and better read than I am in the topic can answer your question. I do not feel as if my answer does it justice completely since I have not read many novels released at the same time as Don Quixote but I have read many of the plays released at the same time.

EDIT: Foucault also writes about Don Quixote in his book The Order of Things (I hesitated to bring him up in my answer earlier because I couldn't remember it for sure although I knew Foucault referenced Borges because my undergrad thesis was on Borges). Anyways, link is here if you're curious.

Let me link a similar thread