There aren't records of ancient peoples experiencing the effects of radiation poisoning, let alone connecting them to specific substances. The radioactivity of natural deposits is generally undetectable without instrumentation. Uranium is actually fairly safe to be around, so long as it isn't in a form where it could be easily aspirated or ingested, like powder. This is due to the fact that Uranium decays through the emission of Alpha particles, which can be blocked by thin materials like human skin. Note that this does not apply to Uranium ores, which also contain decay products like radon that are health hazards in their own right.

If you are interested in learning more about the history of human interactions with radioactive materials, u/restricteddata's reply to a similar question offers more detail on the discovery of radioactivity in the modern era: https://www.reddit.com/r/AskHistorians/comments/401k3n/is_there_historical_reference_to_radiation_or/cyrxaif/