It, not surprisingly, refers to the color.

In the 1800s, the British government required a classification called a Cognac Pale. Essentially, this Cognac wasn't cut with sugar or caramel to recreate the amber colors of finer Cognac. At the time, it was very common for Cognac to be quite distorted from what we expect today.

As for the placement in the name VSOP, the term slowly grew over the years to also require the youngest brandy in the Cognac to be 4 years old at a minimum. Naturally, this fit into the original Cognac Pale standard so the name stayed as other regulations developed.

Cognac: A Liquid History by Salvatore Calabrese

Brandy: A Global History by Becky Sue Epstein

Edit: If you have any other questions about spirits and alcohol, please feel free to ask in more threads. I never get to contribute anything to one of my favorite subs on the one subject I'm comfortable offering comments on.