I'm American, and I've heard mentions of these incidents occurring, but I'm not sure how frequently this actually happened. I have an Austrian friend who was surprised when I told him about this, so I thought I'd delve deeper.
Some cursory internet searches indicate that German prisoners were used for farm labor and such, and I imagine that given the very unlikely odds of making it back across the Atlantic without capture that it was commonplace for them to have more freedom of movement than, say, German prisoners in the UK. But I can only imagine the feelings that this would produce for the Black soldiers, sailors, etc. in America who witnessed enemy prisoners being allowed, in certain respects, a greater level of freedom than themselves in their own country, even as they prepared to fight and potentially die for their country.
A recent article in my local paper states that they were made to work outside of the camps, but usually under fairly tight control. It does not mention them being allowed to go shopping and such, but they were allowed church services.
http://www.vermiliontoday.com/local/history-pow-camp-kaplan-during-wwii