Factory workers often wear blue shirts, blue pants and blue jackets as part of their uniform. The color hides grease without being too drab.

Office workers wear white shirts.

As for why "white/blue collar" and not "white/blue shirt" I can only speculate, but I'd guess it started with fact that a shirt will tend to get dirty first around the collar if a person is sweating. Hence only people who didn't sweat at work (office workers) had the luxury of white shirt collars and thus it became a status symbol.

"Blue collar" probably came along as a means of contrasting factory workers with office workers.