This is one of those weird bits of history that I have researched, going back to when my teenage interest in medieval history dovetailed with a passion for woodworking (see what I did there?).

Anyway, I recollect references to 3 types of 'sanding' of wood before the modern era: a sharkskin called dog shark or dog fish, certain silica-heavy rushes (stiff marsh grass) and leather or cloth impregnated with ground stone, perhaps carried with an oil.

As I said my 'research' was in libraries over 20 years ago and I remember finding it in a book about medieval building. I remember this because I was seriously stoked at digging something out of the stacks. Sure enough, tonight google found it for me in a reference to a reference. In this medieval terms reference book there is a word 'hundysfishskyn' in Middle English Dictionary from University of Michigan, and that points to the book I actually remember looking at: Building in England down to 1540: a documentary history by Louis Francis Salzman. Hundysfishskyn is houndfish or dogfish. Unfortunately Google doesn't seem have this book scanned in and available to look through. But I found another web reference to the text, take it for whatever it's worth to you:

L.F. Salzman in Building in England down to 1540: A Documentary History (Oxford: Oxford Univ. Press, 1952) mentions "sanding" using rottenstone, scouring rush (aka equisetum, horsetail fern, shave grass, etc.), or dog-fish skin. On the latter, Salzman notes receipts for "hundysfishskyn for the carpenters" (Westminster, 1355) and "j pelle piscis canini pro operibus stall" (Windsor, 1351).

As it turns out Equisetum is known for scouring.

So that's a 14th century reference. But wait, there's more!

In the same obscure web site above, there is a reference to the book On Divers Arts: The Foremost Medieval Treatise on Painting, Glassmaking, and Metalwork by a certain Theophilus of the 11th century. This book is partially searchable and seems to turn up the reference to the same rush called shave grass for sanding wood; alas, Google does not preview anything but a snippet.

Incidentally, smooth polishing of fine wooden stringed instruments is still done the same today as it was in the baroque period: with rosen powder after shellac coats.

The above would all be final finishing. After hand planing surface imperfections can be removed with a cabinet scraper. If you've used a scraper before, you'll know the burr on the edge will give you a surface finer than sanding as it cuts the grain, not grind it down. Sanding 'fuzzes' the raw grain and so it's preferred for finishing coats that have stiffened the grain enough for the sandpaper to 'cut'. Scrapers are very old technology: basically a thin piece of steel with some flex. Some handplanes in museums carry confused labeling and are actually devices for holding the cabinet scraper: using a cabinet scraper for extended periods will burn your fingers. I have seen examples if cabinet scrapers dating back to the baroque period.

I hope this trip down memory lane for me helps you.

Edit: some stuff about scraping

I apologize if this doesn't meet the standards of this sub, but another way to smooth wood is to use a tool called a hand plane. They work by removing the topmost (rough) layer of wood and exposing a smooth layer underneath. With certain well-maintained planes it's even possible to have incredibly impressive results.