The idea of exercise and physical activity for no other reason than fitness goes back to the 19th century in Germany. It is Father Jahn who is considered the father of modern physical fitness. He began his endeavors out of a nationalistic spirit from what he saw as the humiliation of his native land by Napoleon. Jahn conceived the idea of restoring the spirits of his countrymen by the development of their physical and moral powers through the practice of gymnastics. The first Turnplatz, or open-air gymnasium, was opened by Jahn in Berlin in 1811, and the Turnverein (gymnastics association) movement spread rapidly.

Goodbody, John (1982). The Illustrated History of Gymnastics. London: Stanley Paul

Edit: Several people are challenging my contention, but I think they are failing to read the OP's question. He asked "when did people begin to exercise for the sake (emphasis mine) of staying fit." That is, when did people begin to work out for the purpose of working out. Or, when did people begin to exercise for the end goal of "being fit." That is the modern concept I am addressing. Of course knights exercised, or swordsman, but they had a specific goal in mind for why they trained. To put it another way, if an athlete is working out in ancient Athens for the purpose of the Olympics, is he simply working out for the sake of working out? No, he is training with a specific purpose in mind.

We have extensive evidence of gym training in Ancient Greece, as described on this previous thread. This may not be the oldest, but it is certainly a direct ancestor of modern fitness culture; our word "gym" comes from Greek.

Edit: OP, could you clarify whether you are referring to exercise in general, or only to exercise by the common people for self-improvement? Greek gyms might not meet the latter standard since a typical non-landowning worker wouldn't have time to attend them.

You may be interesting in the 'Did people in the past exercise/work out/lift weights?' section of our Popular Questions pages, as found in our sidebar.