Mostly elites, whether religious or civil. However, there was a sizeable reading population in some parts of Europe - this is due to the presence of Church schools and the Carolingian education system established in France. A big part of why people think that not a lot of people read in the Middle Ages is because of different types of literacy, and how those types of literacy were presented by those people whose writing we have, who were biased towards a certain type of literacy - Latinate literacy. EDIT Modern historians point to varying degrees of literacy, such as Charles F. Briggs talking about "pragmatic literacy" (Briggs, "Literacy, reading, and writing in the medieval West", Journal of Medieval History, 26(4), pp 400 - 401). See also Franz H. Bäuml, "Varieties and Consequences of Medieval Literacy and Illiteracy", in Speculum, 55(2). Deborah L. McGrady writes of a "vast appetite for books" in the south of France, which she positions as having begun in the twelfth century (Controlling Readers: Guillaume of Machaut and his Late Medieval Audience)

Basically, there were people fully literate in Latin, people fully literate in the vernacular, and people who weren't literate at all - there was a wide spectrum of literacy in most societies, and only one type was considered "true" literacy: Latinate literacy. And so these people are often brushed aside as being illiterate, because they were by the standards of the churchmen writing - they didn't have the same level of Latinate literacy as the churchmen. EDIT The Waldensians, for example, had copies of scriptures printed into the vernacular so they could preach from them, according to Bernard Gui (Practica inquisitionis, c. 1324), but were condemned by Gui for being illiterate in the same text - "...they barely understood [the scriptures]...as they were of little education..."

Lots of heresies towards the end of the Central Middle Ages were literate heresies - Inquisitorial records talk of books being seized and as being central to the spread of these heresies, but in the same breath declare those heretics as "illiterates" due to their lack of Latin. EDIT Examples of these heretics having access to books can be found in Emmanuel le Roy Laudrie's analysis of Inquisitorial records in Montaillou, and in Ginzberg's The Cheese and the Worms, which deals with a miller who had an extensive library.

I'm away from my sources atm, but will edit this comment with some in a few hours.