The ivory would have come from Libya. There were still elephants in North Africa at this time.

The statue of Zeus was constructed inside the great temple at Olympia in the 430s BC. The statue was designed and built by the Athenian sculptor Pheidias, whose previous work included the famous chryselephantine (gold and ivory) statue of Athena in the Parthenon at Athens.

Several later authors describe the statue and the effect it had on visitors to Olympia. Pausanias, who wrote a travel guide to Greece in the 2nd century AD, goes into most detail (5.11.1-2):

The god sits on a throne, and he is made of gold and ivory. On his head lies a garland which is a copy of olive shoots. In his right hand he carries a Victory, which, like the statue, is of ivory and gold; she wears a ribbon and—on her head—a garland. In the left hand of the god is a scepter, ornamented with every kind of metal, and the bird sitting on the scepter is the eagle. The sandals also of the god are of gold, as is likewise his robe. On the robe are embroidered figures of animals and the flowers of the lily.

The throne is adorned with gold and with jewels, to say nothing of ebony and ivory. Upon it are painted figures and wrought images. There are four Victories, represented as dancing women, one at each foot of the throne, and two others at the base of each foot. On each of the two front feet are set Theban children ravished by sphinxes, while under the sphinxes Apollo and Artemis are shooting down the children of Niobe.

He goes on to describe the pedestal, the screens surrounding the throne, and the temple surrounding the statue.

No ancient source explicitly tells us where the material to build the statue came from. However, since the work was created in the late 5th century BC, there is really only one possibility. The presence of the Persian Empire prevented contact between Greece and India, of which the Greeks were only dimly aware; as far as we can tell, the Greeks had no knowledge of the peoples of Sub-Saharan Africa. What they did have was an active trade network with settlements and trade partners all through the Mediterranean and the Black Sea. The gold used for the statue may have been mined on Thasos or in Thrace, but it could also have been imported from Cyprus, Kolchis (modern Georgia), or Spain. The ivory must have come from the only place that had elephants and wasn't hostile to the Greeks - namely, Libya. It is possible that the Greek settlement at Cyrene (near modern Benghazi) acted as a conduit for the trade.