I've spoken briefly about this here, but it depends a lot on how early you want to look, and where. As early as the late Paleolithic, for example, we find carved amber jewelry and amulets, whereas we also have petrified 'snakes' (ammonites) from St Hilda and crinoid-stem rosaries from St Cuthbert much later.

Adrienne Mayor's The First Fossil Hunters is the book on the question, so I definitely recommend you check it out!