In Thoreau's Walden, his account of living by himself in a small shack in the woods by Walden Pond near Concord MA, he tells how he bought a grinding wheel. When the neighbors learned of it, periodically one of them would stop by for a chat...and Thoreau noticed that, when that happened, it was usually the case that the neighbor had an axe to grind...which is the origin of the phrase. That, of course, was a grinding wheel that was pedal driven, perhaps had a trough for water, and so was good for sharpening a very dull axe.

But a whetstone is a far simpler thing- essentially, a flattish piece of sandstone. Besides knives, there would have been axes, sickles, scythes, needles, on a farmstead or in a village. Whetstones seem to have been very common trade items quite early on- archaeologists have found many whetstones for 8th.-9th. c. Scandinavia. Did everyone have one? Like in Thoreau's neighborhood, no one would need a whetstone all the time, so perhaps not. But it seems that with so many it's quite unlikely that their use would have been just limited to blacksmiths.

Baug, I., Skre, D., Heldal, T., & Jansen, Y. J. (2018). The Beginning of the Viking Age in the West. Journal of Maritime Archaeology, 14(1), 43–80. https://doi.org/10.1007/s11457-018-9221-3

As /u/Bodark43 suggests, whetstones are a fairly regular find in Early Medieval and Medieval contexts. Here is a particularly fancy example from the collections of the National Museum of Wales, but outside of ceremonially deposited examples, they're also a relatively common find: The Portable Antiquities Scheme - which typically records small metalwork finds discovered by metal detectorists, field walkers and members of the public etc. - nonetheless has a corpus of over 460 whetstones found across the British Isles. And these will represent just a surviving, identifiable, fraction of a proportion of whetstones that were accidentally lost or discarded and, for whatever reason, never recovered. This, for example, is a nice example of a probable Early Medieval whetstone found recently in the East Midlands as a chance discovery while digging a metal detecting find.

What this broadly shows is that, while whetstones might not have been ubiquitous, they were likely commonplace among the general population.