In short, they were incredibly hard and expensive to transport. In fact, even today only around 14% of grapes are exported due to their highly perishable nature. source

Traditionally, European grape species were only grown between the latitudes of 20° N and 51° N in the northern hemisphere, occasionally reaching 54° N prior to the "little ice age". source Although grape vines are highly adaptable, their ideal growing situation is quite specific; a sloped field, with warm, dry summers and mild winters, and ideally a nearby body of water to stave off the worst of the winter chill. This, paired with the fact that grapes could not easily be transported, meant that unless you grew your own on your own land, acquiring fresh grapes was not necessarily going to be cheap, when it was much more profitable and easier for the grower to ferment them into wine and sell them on that way.

There is also, potentially, a link to the god Bacchus, or Dionysus, who was linked to the grape harvest, and therefore wine, but was also a god for seduction, debauchery, and fertility. It could well be that decadent imagery of a man being fed grapes by his scantily-clad handmaid is more linked to these ideas than any real economic reason.