Fertility has to do with more than just sex. Sex does fall under the umbrella of fertility.

Fertility has to do with procreation, and abundance. This could be herds of sheep, and the multiplication of the flocks. This could be the bountifulness of a piece of ground and how much food could be grown in it. This could be the rains that nourish the fields. This could also be sex, pregnancy, and human procreation. Fertility is all of these things. Fertility goddesses are in the forms of women and women's "fertile parts" (breasts for nourishment, genitals for multiplication), but there were also male fertility gods, such as the Philistine god Dagon, a god of grain (fertility) and fish (multiplication). The central nature of all of these things is the concept of "fertility", or the ability to multiply to produce more of one's species. Humans rely on this force or ability, and should it fail, food sources die and human populations shrink. Ensuring fertility means ensuring the future.

In short, it's much more than just sex, but sex is part of the overall umbrella of fertility.