The term Egyptian is refers to gypsies (there was originally a widespread , though incorrect belief that the Roma gypsies came from Egypt).

It was originally an offence to keep company with gypsies or impersonate them (for instance by offering palm reading services) - source