Edit: Added some reference.

To begin, the rapier is a weapon by-and-large optimized for use in personal defense and dueling against unarmored foes - thus, its light, fast, thin blade. For a wealthy individual trained in its use, it was suitable for protection (or the appearance thereof) while out and about in town. This was the rapier's primary use: fashion statement, personal protection, and occasional dueling implement.

That said, the rapier WAS used in warfare throughout its life. Remember that around this time (mid-16th to late 17th century), the use of Armor in warfare is also declining (though other contributors here can add much more than I to that point by way of both illustration and exception).

Early writers of Italian swordsmanship such as Lippo Bartolomeo Dardi (c. 1415, sadly, no works survive) and Fiore dei Liberi (Fior di Battaglia, ~1408ish) frequently discuss tactics for opponents of mixed protection and weaponry, as well as unarmed tactics. Fiore in particular discusses a huge number of weapons along side the rapier ("spado da lato," in Italian), including polearms, smallswords, and spears. These works can and should be interpreted as military training manuals for practical application, and were clearly taken and used by their contemporaries.

Later writers, such as Salvator Fabris (Scienza d'Armi, 1606) emphasize the rapier on its own, but even then include techniques for handling unarmed foes or foes armed with a lone dagger -- clearly still applicable in personal defense, but nonetheless becoming more of a dueling/personal weapon.

Likewise, we have clear examples in the historical record. For example, French Musketeers were a very real military unit that, while equipped with muskets (thus the name) and used in a variety of roles, were arguably most notable (in both fact and fiction) for their use of the rapier.