I'm trying to find the scene in question but in the mean time, anyone with a Hulu+ account can view the entire film here. It starts at roughly 1 hour and 23 minutes into the film. In brief: to say that the scene greatly understates the atrocities of the Nazi concentration camps would itself be an understatement.
Nazi Germany had established concentration camps as far back as 1933. Chaplin also stated in 1974, "Had I known of the actual horrors of Nazi concentration camps, I could not have made The Great Dictator."
Here is how the film displays concentration camps:
I think the problem here is confusion about what various kinds of camps. Concentration camps were originally a place where you concentrate a (political) enemy - a special, impromptu prison. Mistreatment would be common but not the rule.
Sometimes they would be turned into forced labour camps, which were a source of cheap production power and often a way to do away with people as well.
Finally there are death camps, which were specifically designed to kill people. The most famous one is Birkenau, which was one of the three camps of Auschwitz.
People would have been familiar with concentration camps, but only as seen in the film - a prison for people locked up for political reasons. Spain, the US and the UK had all employed these at the end of the 19th and start of the 20th century, so Chaplin likely drew from those examples.
Concentration camps were used by many nations a century or more before Nazi Germany. In fact the United States may have invented them. http://en.wikipedia.org/wiki/Internment
I took film history, and so I can tell you that Chaplin definitely didn't understand the full extent of the horrors of the concentration camps. He tried to make an accurate portrayal, but he simply didn't have the up-to-date news and full reports from camps. Also, they didn't really do any of the mass killing or anything quite yet in 1939 (when it was being written/filmed), which started in 1942 (I believe). Regardless, the movie does serve as a very good polemic on the mistreatment that Germany was placing on those it punished (specifically Jews in this case, but there were many others affected). If you watch the final speech at the end, this serves to basically tell the entire point of the story.
To get to actually answering your question, his sources would mainly be newspapers, but it's very hard to get to the heart of what went on in these camps at the time, mainly because there were no journalists going to them and few people escaped to tell the tale. He had a better understanding of the ghettos, where a portion of the film resides, so that section of the movie is more accurate.
