Has there been documented instances of people identifying as neither male or female before the 20th century? If so, was it common and how was it 'treated'?

There are! The most prominent I can think of is gender identities in India. The first mention is in the Kama Sutra, compiled before the 3rd century AD, which actually refers to a third sex, the tritiya prakriti, interpreted as likely being transsexuals.

The existence of genders outside the binary is actually prevalent throughout Hinduism. When Lord Shiva combines with his wife, Parvati, he becomes Ardhanari, the patron saint of Hijras (Hijra is a sort of catch-all term for transsexuals, transvestites, and eunuchs). In fact, Shiva himself frequently changes fluidly between male and female, though it's unknown whether this was to represent gender ambiguity, or if it's from the fact that Hinduism is a very, very diverse religion, more like a family of religions, that has little consensus on many topics.

In the Ramayana, there is a part where Rama goes into a 14-year exile. A crowd of devotees follows him, and he gathers them in a forest and tells them that all the men and women should return home and take care of themselves. After his 14 year exile, when Rama returns, he finds a group of hijra remaining in the spot he told them to leave, for they considered themselves neither man nor woman.

These examples are not historical, but they are prevalent enough in major aspects of the culture in a way that suggests that people were very aware of sexes and genders existing outside the binary. Various Buddhist texts acknowledge them as well, with some saying it's natural and that gender only exists in the physical world, while others view them as abominations that are lustful and only seek to satisfy their physical urges.

For a historical example, one needn't look further than the Criminal Tribes Act of 1871. It was initially created to curb violence by criminal organizations/tribes, most prominently the Thuggees. Thuggee is where we get the word "thug" from, and estimates of their crimes have put them at #1 for most destructive terrorist group. It was also used to criminalize certain types of people the British viewed as dangerously, unseemly, or otherwise depraved, such as gypsies. One of the groups criminalized was Hijras. This allowed the British to carefully monitor and regulate their behavior.

For further reading, there is a paper on Hijras in Imperial British India called "A Right to Exist: Eunuchs and the State in Nineteenth Century India" by Laurence Preston. Maybe someone with more expertise can chime in on how trustworthy Preston is as a source, he seems to have a few academic writings on various topics on modern India.

tl;dr Yes, non-binary genders have been a mainstay in South Asian culture for millennia.