The first Olympic games was held there

What do you mean? The Olympic festival was always held at Olympia in antiquity. That's why they're called that, they're named after Olympia. The degree to which the Elean state controlled the site of Olympia was contested over time, but the festival was always held at Olympia. The habit of moving the festival around to a different city every Olympiad is a purely modern convention