There is a scene in the aforementioned movie in which this conversation takes place.
"She's a beauty, my lord, as would take a king to church for the dowry of a nutmeg."
"My plantations in Virginia are not mortgaged for a nutmeg. I have an ancient name which will bring you preferment when your grandson is a Wessex."
What, in this context, is a nutmeg? I know it is a spice which was popular during this time but I can find nothing anywhere to explain why the word is used to describe a woman.
I checked the Shakespeare Lexicon and Quotation Dictionary (Volume 2, pg. 785) for "nutmeg". I found three occurrences, but they aren't very helpful to your question:
CLOWN: ... I must have saffron to color the warden pies; dates, none - that's out of my note; nutmegs, seven; a race or two of ginger - but that I may beg; four pound of prunes, and as many of raisins o'th' sun.
DUKE OF ORLEANS:
He's of the colour of the nutmeg.
DUMAIN:
A gilt nutmeg.
However, the Lexicon notes that nutmegs were widely available and commonly given as Christmas gifts in England. So in this context, a nutmeg is something that is common and of little value.
The "dowry of a nutmeg" line is suggesting that Viola is so beautiful that even if she had a tiny dowry, she would still be a desirable wife. Lord Wessex's response, "My plantations in Virginia are not mortgaged for a nutmeg." tells us that it is the dowry and not the beauty that he is interested in.