I've written on this topic a few times before, so I'll post one of my older answers below this
You seem to be under the impression that the average viking raider even had an expectation of Valhalla if he died in battle and that this belief was widely shared among the Norse culture at large. I can't really blame you, many misconceptions about the Norse have seeped into popular understandings of the Norse world that have little basis in academic consensus. Neither of these assertions, that warriors expected to go to Valhalla or that the average non-warrior did, are really on solid ground however. We quite simply do not know what the average beliefs of the Norsemen were prior to Christianity. We don't even know if all of the Norse worshiped the same set of deities, and it seems overwhelmingly likely that they did not. Simply looking at the deities invoked in place names gives you the idea that certain deities were more popular than others in different parts of the Norse world and at different times, and in ways that do not match up with more popular understandings of the Norse people.
This is the first step towards understanding that Norse religious practices and beliefs were not uniform either in time or space. What a Norseman might believe if he was a raider in the 9th century was different from what a farmer might believe in the 11th, and in turn both would be different to what a trader living in the lands of the Rus might believe in the 10th century. Our modern western (ie Judaic/Islamic/Christian) understanding of religion gives us a certain expectation of what religious life should look like, but this expectation need not accurately map onto history.
There never was one single "Norse religion" that was doctrinally consistent over the Norse/Germanic world, either temporally or geographically. The stories that Snorri Sturluson edited and compiled into his own works almost certainly were not the same as the stories that held sway in Sweden, or Geatland, or Saxony before its conquest by Charlemagne. Indeed Snorri's own work was compiled centuries after conversion to Christianity in Iceland, long after remnant communities would have stayed pagan. Indeed, even the Eddas are inconsistent on who gets to go to Valhalla or Freyja's Halls, many sources make no mention of Freyja's halls at all, and this ambiguity and inconsistency shouldn't be surprising.
This inconsistency in the sources seems to indicate to me at least, and certainly plenty of scholars who have fancy degrees and DO read Old Norse, that there was never any sort of doctrinal coherence to Germanic paganism or Old Norse practice. So this is a roundabout way of saying that while your question is a reasonable one and certainly an interesting one, it unfortunately will likely remain an unanswerable one.
Perhaps it might be best to conclude on an analogy. I do not know your particular religious affiliation, but I'm going to assume that you're roughly familiar with Christianity. Christianity has many things that Norse paganism lacks, such as a single coherent book from which the majority of the religion's theology is derived, and yet any conversation with different denominations or a cursory examination of religious history will show that getting everyone to agree what constitutes Christianity and what the various beliefs of the religion should be is extremely complicated. Hell, theological debates, excommunications, and the like have raged between old and storied Churches over a single word in a prayer. Now take all of those divisions, and remove the Bible as an authoritative source. Now imagine what that might mean for the religion. Indeed, it might seem that it would be all but impossible to construct a religious system with firm answers to a lot of questions in the absence of such a central work.
How can this be you might ask? Norse mythology is almost as well known these days as Greek mythology, surely we have an extensive source basis to rely on such as the Sagas? Unfortunately however, we cannot treat the sources available to us like we would religious texts such as the Bible, Talmud, or Koran. The only literary sources on Norse religion come from outsiders, ie Adam of Bremen and Ibn Fadlan, or are centuries removed from the actual practice of Norse religion, ie the Sagas. The Sagas in particular come in for a hard time when it comes to their accuracy in regards to Norse religious practices. They were compiled in a specific cultural context (13th century Iceland) and for specific aims (to legitimize and glorify extant Icelandic families for the most part), and not to accurately retell the beliefs of the pre-Christian Norse. Indeed Iceland had become Christian some two centuries before the Sagas were written down. For reference, the United States has existed for about 250 years, and religious expression has changed dramatically in the intervening centuries. Imagine how religious practice might be different in the absence of what we take for granted in religion today such as professional clergy, written scriptures, dedicated houses of worship, and so on.
What does this mean for your question though? Mostly that all of our "common" conception of Norse beliefs are not really applicable to the Norse people as they actually existed. This applies to Valhalla as well. While it might be unlikely that Snorri Sturluson invented Valhalla completely as a part of the Saga compilation, much of his work is infused with Christian influences already, the importance of Balldr as a sacrificial/redemptive figure for example. Did a Norse warrior in the 9th or 10th centuries have an expectation of going to Valhalla? Did the people who lived in Scandinavia as farmers despair over the fact that they would never enter Valhalla? Its really impossible to say, but I find it unlikely they spent a good deal of time worrying over it. Our understanding of Norse religion is incomplete, and we will never really be able to reconstruct the belief system of the Norse in all of its diverse manifestations.