While I'm not sure if any group ever used the adjectives "evil" or "unnatural" in reference to the Sangha, John Keay's China: A History lists Buddhist celibacy as a reason for the long period from introduction to adoption in China. China's primary philosophy for centuries was Confucianism which placed a high value on "filial piety" specifically calling attention to paying proper homage and reverence to one's fathers. So the idea of abandoning family and lineage was fascinating, but abhorrent to the first Chinese scholars to be introduced to this foreign idea.

Within India where Buddhism originated, celibacy has had some form of religious practice and orthodoxy by the time of the Vinaya (Rules for Monastic Conduct). The Buddha's life story usually includes his time in a wilderness when he deprived himself of life's pleasures (and even necessities, keeping himself from food and water for long periods of time). He was reportedly not the only one to do this. He engaged in this practice with the sramanas, the ascetics, who believed (more or less) that increased suffering in this life led to an inevitably better rebirth. The first followers of the Buddha were sramanas, and this is the word used to describe novices in the Buddhist sangha.

tl;dr, there is a long pre-Buddhist history of celibacy in Indian religions that Buddhism merely adopted for its own purposes (the elimination of desire). Before Buddhism became accepted in China, it was rejected because the notion of abandoning family and lineage was directly opposed to Chinese philosophy, specifically filial piety.