Wikiquote claims that it's found on p. 113 of a 1965 book called The Mystery of Matter, a collection of essays on physics and whatnot edited by one Louise B. Young. It isn't fully accessible on Google Books — I can only get it to give me a cropped image of page 113 — but that does show that at least part of the quote is there. No way to tell if it's cited in any way, though, at least not without getting my hands on a hard copy.

The site AZQuotes claims that the quote comes originally from Aristotle's Metaphysics, but I can't find any of the relevant words in that text (using the Perseus text search at Tufts).

Those are the only two web references I've found that attempt to cite a source. None of Pythagoras's own writings are known to survive, and even his life is shrouded in mystery and myth. I personally suspect that the quote is spurious, and it's attributed to Pythagoras because it sounds a bit like something he'd have said.