EDIT 2: This is a new, hopefully more sensible answer. I have kept the first one below for context.

The mechanisms that make bicycles so easy to balance are surprisingly intricate, and are not fully understood even today [Wilson, p. 263-]. To get an idea of the principles at work, look at this video by Dr. Arend Schwab, who specializes in bicycle dynamics. The ability to steer to correct the position of the center of mass is certainly key [MIT]. You cannot ride a bicycle if you fix the handlebars.

Incidentally, Schwab's lab maintains a bibliography of the study of bicycle dynamics. The first paper mentioned there is from 1869. This was before the current design with chain-driven rear wheels, called safety bicycles, appeared around the 1870s [Herlihy, Part 4].

Modern studies of bicycle stability such as [Meijaard et al] point out several flaws in earlier studies that were nevertheless considered authoritative. The sophistication required to fully account for the stability of bicycles makes it unlikely that von Drais, the inventor of the first proto-bicycle, could have predicted its stability on theoretical grounds. In this sense he did not previously know that it would be feasible balance oneself on two wheels. For instance, the mechanisms which explain the stable dynamics of some simpler systems such as rolling hoops (those have been around at least since Antiquity), are not sufficient to explain why it is so easy to balance on a moving bicycle [Jones, p. 51].

Karl von Drais's (link in German) 1817 Laufmaschine is widely acknowledged to have been the model for the original bicycles. His design did not include pedals, so you had to propel yourself with your feet, although he did demonstrate that he could balance himself on two wheels while rolling down slopes. [Lessing] points out that von Drais was an ice skater, and argues that this might have led him to the idea that balancing might be possible if you can shift your center of mass through forward motion. This mechanism is also at work when ice skaters balance on one skate.

Sources:

[Herlihy] Herlihy, David V., Bicycle - The History, Yale University Press, 2004.

[Jones] Jones, David E.H., The stability of the bicycle, Physics Today, April 1970.

[Lessing] Lessing, Hans-Erhard. Cycling or Roller Skating: The Resistible Rise of Personal Mobility, Proceedings of the 5th ICHC, 1995.

[Meijaard et al] M.P. Meijaard, Jim M. Papadopoulos, Andy Ruina, A. L. Schwab, 2007 ``Linearized dynamics equations for the balance and steer of a bicycle: a benchmark and review'' Proceedings of the Royal Sciety A 463.

[MIT] Why is a bicycle easier to control when it's moving?, MIT Ask An Engineer.

[Wilson] Wilson, David Gordon, Bicycling Science, 3rd Edition, MIT Press, 2004.

---

EDIT: This is not a good answer, see below.

Although I do not know much about the history of bicycles, I will make the following comment: what keeps you upright when you ride your bike is conservation of angular momentum. Although the modern concepts of momentum and inertia were only fully clarified in the 18th century, Newton's derivation of Kepler's law of areas in Book 1 of the Principia is essentially equivalent to what we now call the conservation of angular momentum. Book 1 appeared in 1687.

Regardless of the theory, angular momentum conservation is also what keeps tops spinning upright and it is the principle behind gyroscopes. Most sources I could find indicate that the first bicycles originated in the 19th century. Both spinning tops and primitive gyroscopes predate the 19th century.