The lance fell out of use as a result of improved infantry tactics. Around 1300, the pike came into use to defend against the charge of heavy cavalry. Infantry-based armies in Scotland, Flanders and Switzerland were able to defeat cavalry-based armies.
Gradually over the course of several centuries, armies came to rely less on cavalry and more on infantry working in a tight formation. The appearance of firearms on the battlefield increased this trend, as even heavy armour could be defeated at longer ranges (though late plate armour did offer a measure of protections against the firearms then in use).
These two trends led to the creation of the pike-and-shot formation, consisting of pikemen and musketeers. The most notable example are the spanish tercios. These formations were - properly deployed - able to defeat cavalry charges with ease.
As a reaction, cavalry did away with most of its armor and the cumbersome lance, leaving a breast plate and a helmet. Cavalry also adopted firearms (pistols) as their main weapon and devised tactics to use them (caracole). The lance had disappeared from the battlefield.
This in turn contributed to the demise of the pike as well. Without lance-armed cavalry, the major enemy the pike was designed for had disappeared. The tight formations developed to maximise the protection offered by the pike proved to be tactically inflexible and also vulnerable to the field artillery that appeared during the 30 years war. For example, Wallenstein dropped the tercio organisation at the beginning of his second generalat, as it had proven to be inferior to the greater tactical flexibility of the swedish forces under Gustav Adolf. Lastly, the development of the bayonette meant that every musket could double as a pike, and pikemen were rarely seen on battlefield after 1700.
This - once again - opened the possibility of a heavy cavalry charge. A lance once again offered a range advantage over a musket with an affixed bayonette. Napoleon proved that a charge by lance-armed cavalry was capable of delivering a greater shock than that of sword-armed cavalry, and several other powers imitated him.
Ultimately, the increasing firepower and range of infantry weapons made cavalry charges ineffective.