Literature as an entity distinct from religious tales, myths, moralistic fables and even cultural commentary is a specifically Western and very recent phenomenon. Its roots go little further back than the late 18th century. An excellent book about this development is The Literary Absolute by Jean-Luc Nancy and Philippe Lacoue-Labarthe. The authors trace the genealogy of this new, absolute notion of literature from the group of poets and writers (such as Novalis and the two Schlegel brothers) based in or around the University of Jena in pre-unification Germany. Remember that nationalism is on the rise in Germany and in Europe in general around this time. The French Revolution begins in 1789. This is the backdrop against which this new, autonomous notion of literature and distinct national literatures emerges. Under Napoleonic rule, German literature and language become the only viable recourse to sovereignty for German nationalists, and while it sounds absurd, this little crisis in German political life gave us the idea of literature as a distinct discourse independent of sociology, history, politics, philosophy and others.

The other chapter in this story deals with British colonial history. Remember that no major universities in the world, including Oxford and Cambridge, offered degrees in Literature or Literary Studies at the beginning of the 19th century. The colonial project of comparative philology, to which we owe our understanding of the various language families today, initiated not only translations of literary works from various cultures into several European languages, it cemented the category of literature itself as something distinct from its linguistic and cultural context but also a window unto that context. Literature thus became a means to understand and appropriate the culture of the colonised. Many works of native high culture were deemed representative of the culture as such, with little regard to the actual lived practices or the political and social cultures of the vastly diverse people that the British empire needed to govern over the years. The scale of governance required was a major concern as the empire grew. Translations were undertaken on a massive scale and grammars of several largely unwritten languages produced. The East India Company pioneered this use of elite, high literature as a tool for simplifying vastly diverse cultures into administratively manageable entities, and the British empire inherited this from the Company when it took over India in 1857. (As an aside, the US inherited this attitude in the guise of Area Studies departments that still operate with these implicit assumptions about the global south.)

In short, the category of (fictional) literature is a fairly recent Western phenomenon born out of the contradictions of European nationalism and global imperialism. This is not to say that literary works are inherently nationalist or colonialist but our dominant understanding of literature as a discourse is inherently caught up with nationalistic and imperialistic concerns. It says more about the cultural (educational) and corporate (publishing) institutions that have grown around literature than it does about how you or I can read fiction. Reading is at its most radical when it ignores or goes against institutional dogma. Enough of these institutions' role is to filter and keep the most radical aspects of literary writing in check. Which is also the context in which Brother Mouzone makes that remark in The Wire about 'a nigga with a library card' being the most dangerous thing in America.

Sources:

The Literary Absolute by Jean-Luc Nancy and Philippe Lacoue-Labarthe

Orientalism by Edward W. Said

Masks of Conquest: Literary Study and British Rule in India by Gauri Vishwanathan