The color "green" has been associated with radioactivity because of the public associations with radium, which was often experienced as having a greenish-yellow glow to it, especially as radioluminscent paint. So green = radioactive has been a cultural association since the 1910s or so (even when in many contexts blue = radioactive would be a more sensible association, as an aside).

Separately, it is worth noting that uranium ore comes in many different appearances depending on how much uranium content, and how many other things, are in the ore. Low-grade ores like those from the United States Southwest, which are often what one sees in depictions of ore, are a dusty yellow. Sometimes this is how uranium is depicted in popular culture as well, especially in materials from the US "uranium boom" of the 1950s (when the US government set an artificially high price on uranium in order to encourage prospecting). I don't think this is too far from "green" myself, but to each their own. "Yellowcake," the separated uranium oxide, is yellow, though there are other separated forms as well in different colors (e.g., "green salt," uranium tetrafluoride).

High-grade uranium ore, such as those which come from the Congo, can indeed be green — very very green. Again, the coloration depends a lot on the uranium content and the other minerals. Uranium when turned into glass (as was popular in the 1920s) is again a pale yellow-green, and under black light is bright green.

(Uranium metal is absolutely not green at all.)

Anyway. The long and short of it is: sometimes it is quite green, though the popular depictions of "radioactive = green" descend from associations with radium, not uranium.

On nuclear imagery in general, Spencer Weart's The Rise of Nuclear Fear is especially good, and tracks public associations with radioactivity over the decades, both prior and after the atomic bomb.