It's complicated to generalize for all Western societies, but yes, you can certainly argue for the existence of non-binary genders in different European cultures prior to the 19th century. In fact, you can kinda go the other direction and argue that Europe only recognized one sex and multiple genders! The one-sex idea is argued in the book Making Sex: Body and Gender from the Greeks to Freud by Thomas Laqueur, which made a splash in gender studies in the 90s, though it's not without critics. The easiest example of a "third gender" is the gender of eunuchs, and where they would have been placed on the one-sex model has been explored by Roger Freitas in this paper. In short, they would have been understood as some sort of subset of the masculine gender, but not fully masculine. Femminielli may also be of interest to you. I am not very familiar with the gender situation outside of Italy and England for the 18th century but this does serve to show that the idea of more than 2 genders has existed in Europe.

How are you defining gender? As in, for you to accept the existence of three genders what factors would have to exist? People thinking of themselves as neither man nor woman? Legal distinctions?

Following from Caffarelli's point, Swanson (ref. below) argues for the medieval clergy in Europe as a third gender for the reasons descirbed in relation to eunuchs. This was not a legal definition- but gender is about identity and society more than legal definitions, surely.

Swanson, ‘Angels Incarnate…’ in Hadley (ed.) Masculininty in Medieval Europe, (1999)