Were there 'colors only' restaurants? Or were there only 'white only' places?

My impression was that whites could basically go anywhere they wanted, they just didn't. And thus the follow-up question.