The absence of a chain drive and gearing.

The humble modern bicycle seems like such a straightforward invention but it really is a technological marvel which is only possible due to a series of technological innovations, the most important being a lightweight high strength metal frame and smoothly rolling bearings for the wheels, those came first and gave us the early bicycles including the penny-farthing, later innovation (the chain drive and pneumatic tires) gave us the bicycle we're more familiar with today.

The very first bicycles were just for coasting and propelling yourself with your feet directly on the ground. Various types of means of locomotion were devised but the most popular became the pedal, which was attached directly to the hub of the front wheel. The problem with this design is it severely limits speed, with a 1:1 linkage you can only go as fast as you can pedal. A person might be able to pedal with a cadence of 60-90 rpm which translates to between 3600 and 5400 rotations per hour. At a bike tire circumference of 8 feet or so (for a nearly 3 foot diameter tire) that's a speed of just 5 to 8 miles per hour, which is just jogging speed or a little more.

But with a larger tire you get more speed (though also more difficulty going up hills) at the same cadence. If you extend that as far as you can go given a person's height and leg length you end up with a penny-farthing style bike with one very large wheel. Such bikes could go much faster, with top speeds in the 20 mph range, similar to the speed of a horse at a canter. The downside being that you have to climb up onto it to start and potentially if you fall off the consequences could be more disastrous. Additionally, the large wheel led to a smoother ride because it could more easily roll over changes in the riding surface without jostling the rider as much, an important feature when tires were made of leather, metal, or solid rubber.

But not everyone was comfortable riding such daunting vehicles while others, especially women, couldn't due to their clothing (since it was not considered "ladylike" for women to wear pants in the late 19th century in the Western world). This led to the invention of tricycles, quadcycles and other pedal driven vehicles using a direct chain drive and the same large wheel sizes as the penny-farthing, making it possible for riders on such vehicles to keep up with groups of cyclists. These became very popular in the 1870s and 1880s but were comparatively more expensive (and harder to store). It should be noted here how much the cycle chain itself is a mechanical marvel, each link in the chain contains two small rollers which serve as a bearing surface against the gear teeth and the whole thing is able to move through the gears with low friction and low wear.

Just before 1880 the first "safety bicycle" was built using a simple two wheeled design with small wheels and a chain driven gear linkage with a gearing advantage that made it possible to reach higher speeds than a penny-farthing. These bikes were partly marketed to take advantage of the reticence of the public toward the perceived difficulty and risk of riding a penny-farthing, and provide a more comfortable and easier riding alternative without the cost and size of a tricycle or quadcycle. It wasn't really until the first practical pneumatic tires were introduced in the late 1880s before the penny-farthing was rendered truly obsolete but already by then the safety bicycle had become immensely popular. For a while penny-farthing die hards held on due to a devotion to the engineering simplicity of the penny-farthing. Ultimately it was a lost cause due to the numerous advantages of the pneumatic safety bicycle. Scarcely a decade later the first Tour de France was run in 1903 with the average speed of the winner coming in at about 16 mph over 94 and a half hours of riding, something that just wasn't feasible (let alone tolerable) with a penny-farthing at those speeds and that duration of riding.

For fun, here's a video of an early safety bike being ridden, and here's a nice little informative piece on such bikes while showing off a hand built reproduction.