This is a bit from my normal expertise, and I can only answer for ancient Greece, but yes, there are a number of credible sources indicating that very large bones were found. It is, to my knowledge, uncertain whether these bones were true fossils (from dinosaurs or other animals) or just large bones of extinct prehistoric creatures.

As one source of evidence, Pausanias, a travel author (sort of like a modern Blue guide) in the 2nd c. AD reports that (Description of Greece 2.10.2) "In the portico lies a huge bone of a sea-monster". These bones are often interpreted as remains of an extinct species.

In addition to this, an early philosopher called Xenophanes supposedly interested in fossils and geology (OCD^4 s.v. Xenophanes).

For more, I can recommend Adrienne Mayor's The First Fossil Hunters: Dinosaurs, Mammoths, and Myth in Greek and Roman Times