Strictly speaking we don't (and can't) know what religious/cultic aspects were applied to the various members of the Vanir in actual practice. So why do scholars and pop culture refer to them as fertility god(esse)s?

The main three Vanir that get mentioned are Niord and his two children, Freyr and Freyja. Let's look at their descriptions in Gylfaginning:

Niord

The third As is the one called Niord…It is to him one must pray for voyages and fishing. He is so rich and wealthy that he can grant wealth of lands or possessions to those that pray to him for this. Niord is not of the race of the Æsir. He was brought up in the land of the Vanir…

Freyr

Freyr is the most glorious of the Æsir. He is ruler of rain and sunshine and thus of the produce of the earth, and it is good to pray to him for prosperity and peace. He also rules over the wealth of men.

Freyja

And Freyja is the most glorious of the Asyniur…She is the most approachable one for people to pray to, and from her name is derived the honorific title whereby noble ladies are called frovur. She was very fond of love songs. It is good to pray to her concerning love affairs.

All of those descriptions would seem to fit the idea of being fertility gods. They are associated with prosperity, peace, wealth, and love. To go a step further, Freyja is associated not just with love but desirability and lust/promiscuity. Male Jotuns attempt to make deals with gods in order to win Freyja. The builder of Asgard is a Jotun who wants Freyja, the sun, and the moon. It is only through the trickery of Loki (and then Thor's hammer) that he doesn't get them. Another giant steals Mjolnir and attempts to do a hostage exchange, the hammer for Freyja. Freyja is accused throughout the literature of being unfaithful and sleeping around. Probably the most entertaining is during the flyting in Lokasenna where Loki accuses her of having slept with every single male in the room.

So what do other texts have to say about them? In Ynglinga Saga, Freyr ascends to the throne at Uppsala and proceeds to accrue wealth and his rule is "blessed with good seasons". After his death he is entombed in a grave mound and the people bring offerings to the mound and the "peace and prosperity" continue. In Haakon the Good's Saga there is an episode at a feast where several mugs of drink are blessed by the chieftain. Odin's is "emptied for victory and power to his king" while Niord and Freyja's mug(s) are emptied "for peace and a good season."

The evidence throughout the literature seems to point towards the Vanir as being associated with the things that fertility gods are always associated with. The archaeological evidence is a bit more scant. There are figurines that have been found that appear to be of fertility deities but there are no inscriptions that I am aware of that outright say "This is a figure of Freyr". So any attribution of a figurine or carving that appears to depict a Norse fertility god is thought to be of Freyr if male or (usually) Freyja if female. That's a bit of a circular argument though and can't really be applied as evidence for the status of the Vanir.

Having said all that, there are those that contest the Vanir being fertility gods. Sadly, I do not have access to the book but in The King, The Champion, and The Sorcerer Lotte Motz apparently argues that based off of "the categories of 1. Agricultural concerns, 2. Homage paid to the Gods, 3. Relation to Wealth, 4. Relation to War, 5. Sexuality, 6. Creativity and Regeneration, 7. Relation to Magic Skills, 8. Relation to Sea Travel, 9. Animals of Association and 10. Relation to Royal Rule" (Rudolf Simek 13) that the standard designation applied to the Æsir and Vanir is inaccurate. It would be more appropriate to classify the Vanir as being associated with royalty and power, while the Æsir should be associated with more agricultural pursuits including agricultural fertility (For example, Odin appears to have closer association with sexuality than Freyr does). Again I don't have access to that book so sadly can't break down her argument further.

In addition to all of that, Rudolf Simek argues that separating the Æsir and the Vanir is an anachronism imposed by Snorri. The term Vanir is extremely rare in the eddic and skaldic poetry. The only time where it is unequivocally used as a term for a separate race of gods is in Snorri's prose. Every other use is much less clear cut and open to interpretation. Simek argues that the concept of the Vanir is a literary invention of Snorri allowing him to tie together some of the players in his subject matter (there is a semi-mythic figure called Vanlandi. Simek supposes that the idea of the Vanir was an attempt by Snorri to link a possible etymology of Vanlandi's name into the euhemerized account of the Norse people's descent from the Trojans). This argument doesn't really affect whether Niord, Freyr, and Freyja should be considered fertility gods but it is an interesting side note to the whole thing.

Anyways, I hope that helps a bit. Sorry that I couldn't be more specific on some of the stuff. Most of the sources I had on hand or tracked down pretty much took it for granted that the Vanir are fertility gods so it was a lot of consulting different, varied stuff to try to piece some of the reasoning together.

Works Consulted

Gods and Myths of Northern Europe - H.R. Ellis Davidson

Haakon the Good's Saga

Heimskringla - Snorri Sturluson

Norse Mythology - John Lindow

Poetic Edda

Prolegomena to a Cosmology of Healing in Vanir Norse Mythology - Valgerður H. Bjarnadottir and Jurgen W. Kremer

Prose Edda - Snorri Sturluson

The Vanir: An Obituary - Rudolf Simek

Verses versus the 'Vanir': Response to Simek's "Vanir Obituary" - Frog and Jonathan Roper

Women in the Viking Age - Judith Jesch