Some do.

Rosemary, for instance, is often associated with fidelity and remembrance. This association is at least four hundred years old, as Shakespeare mentions it in Hamlet (Act 4, Scene 5, while Ophelia is speaking) and seems to have expected his audience to make sense of the words.

In many cases "flowers" and "herbs" don't have to be entirely distinct categories. You can cook with roses, violets, lavender. The "language of flowers" as a defined collective concept with formalised meanings appears in the early nineteenth century, with books like Henry Phillips' Floral Emblems published in 1825 and Frederic Shoberl's The Language of Flowers in 1834. However, individual associations for specific plants (like Shakespeare's rosemary) can be much older, and can include many plants with culinary or medicinal uses.