From what I understand, for most of human history only a select few were literate. As such, most of our primary sources weren’t written common people of the time, but by the elite and well off. They were the only ones educated enough to be able to write down historical events.
Is there a general consensus on how this effects our view of history? How likely is it that the primary sources from which we learn history preclude us from learning about the average person at the time?
You're in very good company with this question! The answers to this question, as well as arguments over how important this question really is, make up a vast portion of the historical debates of the 20th Century. I think there's an argument to be made that Marx is the first historian to grapple with this idea. His theory of historical materialism, or the idea that history is driven by the physical and economic conditions of people, by necessity asked that historians look at the lower classes. The Marxist perspective on history has somewhat fallen by the wayside lately, but the legacy of his ideas has influenced pretty much all of the major theories of history since then.
After Marx, a whole range of historians continued to interpret and build upon his ideas. Some of them are best described as Marxists, including a very influential Italian man named Antonio Gramsci. Gramsci himself was a Marxist political theorist most famous for his theory of hegemony. The theory of hegemony says, more or less, that the ruling classes create systems that promote a single dominant worldview, in order to keep the lower classes from thinking about revolting. This theory was a major development, because it offers a strong argument for why historians have to look very critically at historical documents (especially for, as you say, the parts of history where few people can write and the only documents we have are from the ruling classes).
Others, however, rejected the label of Marxism and fell into other schools of thought. During the 1920s, one such school emerged in France. This school (called the Annales school) took what is now called the "longue durée" view of history, focusing on long-term social trends and structures, and relied heavily on economic information like crop prices and city populations. One historian within this group, Lucian Febvre, coined the phrase "history seen from below and not from above" in 1932. The Annales perspective, similarly to the Marxist perspective, emphasizes the experiences and the actions of the everyman and not just the upper classes who were the most likely to be literate.
In the 1960s, these sets of ideas (along with many others!) formed the theoretical roots of postcolonial studies, or the 'history of the subaltern.' In the history of the subaltern, your question finds its purest expression: what claims to truth can we really make about the history of societies who have had an alien social system and alien modes of thought imposed on them for generations, and their own systems and ways of thinking suppressed or destroyed? Beyond the lack of written sources, the system of colonization tended to take away the types of historical record that the subaltern cultures actually used - their oral traditions, their art, and so on. The colonizers of course wrote their own histories, but what can modern historians do to recover the authentic experiences, beliefs, and so on of cultures that have been largely erased by colonization?
So, with all of this historiography and theory in front of us - how do historians actually do history from the bottom up? One method, which became very popular with English and American Marxist historians starting in the 1960s, was to just go ask people. 'People's historians' like E.P. Thompson in the UK and Jesse Lemisch in the US began to go into working class communities and interview the older people of the community about their past experiences. This sort of oral history has since been done quite well in Holocaust studies, the social side of military history, and other topics involving the average men and women of a society. However, oral history has its obvious limits. People forget things, or feel differently about them at 75 than they did at 25, and of course we can't go back and interview people who are dead.
The Annales school's solution to this was to look at physical things instead. These included the man-made objects of everyday people like spoons and clothing, but also maps, old wells, the position of roads and buildings and farms, and so on. This side of material history can tell us a lot about how people lived and the social structures they inhabited, but it can't tell us how people thought and felt. So, historians turn to the written record, but with a more critical eye. One of my favorite historians calls this "forcing the past to speak, even against its will." But what does this look like?
An example I read recently, which I very much liked, is called The Trial of Tempel Anneke: Records of a Witchcraft Trial in Brunswick, Germany, 1663. It's a reconstruction of a witch trial using entirely the official court documents, but by refusing to accept the narrative of the inquisitors running the trial these historians were able to tease out an understanding of who this accused witch really was. In this process, they also teased out an image of who, perhaps, many other accused witches really were.
I hope my examples and crash course in theory helped answer your question!
Sources and recommended reading:
Bloch, Marc. The Historian’s Craft, trans. Peter Putnam. New York: Vintage Books, 1977.
Green, Anna and Kathleen Troup, ed. The Houses of History: A Critical Reader in Twentieth-Century History and Theory. New York: New York University Press, 1999.
Morton, Peter A. and Barbara Dahms. The Trial of Tempel Anneke: Records of a Witchcraft Trial in Brunswick, Germany, 1663. Toronto, ON: University of Toronto Press, 2005.
NB: I have included several in-text Wikipedia links for the reader to quickly familiarize themselves with some of the more arcane theoretical concepts in this post which they may not have heard of before, but these are not a replacement for good scholarly discussion of these topics.