In Edward Hopper's painting Nighthawks, the men are seen wearing hats at a diner bar.This painting is from 1947 I think. In an episode of The Sopranos, Tony has a younger man take his hat off in a restaurant in what seemed to be an attempt to uphold a standard of decorum in a modern world gone awry. What did it say about the men in the painting that there hats were on? Was Tony's attempt at preserving old values misguided? To what degree did the wearing of a hat indicate class at that time?
Think of wearing a hat as part of being fully dressed. Somewhat akin to shoes today. If someone is walking around without shoes at the beach, it's no big deal. They're probably in casual clothing, maybe a bathing suit. But if the same person walked into a restaurant or the office barefoot it'd be weird. We wouldn't think of him as trying to be rude or throwing aside old values. Just plain weird and not fully dressed.
Removing the hat indoors is done either in formal circumstances (church, formal dinners, etc) or in private locations (home, office, etc). A casual location like a diner sort of exists between the two, not being a private location, but not being formal in the least. Chances are if they were specifically there to meet with someone for a length of time, they would remove their hat. Compare this to your winter coat. You'll certainly take it off for a nice restaurant or when you go into a persons home, but if you're just planning a quick meal at a diner you wouldn't feel like it was necessary to remove it unless you were too warm.
We tend to assume there were all sorts of unwritten and difficult rules of decorum in the past, but in reality most came down to the same way we deal with things today. If you are the only person in the room doing or wearing something, you'll feel out of place. It's easy to pick up on social cues very quickly and follow along. These actions repeated develop into habits that affect similar circumstances. If you grow up going to church and seeing every man remove his hat upon entering, you'll pick up that habit and repeat it at any church you attend.
Wearing a hat doesn't define class, but it can define what job you're doing. Work uniforms compared to a business suit, for example. The type of hat, condition, or fit can say much more. In terms of the issue of the young man not removing his hat, he's probably the type that hasn't been in many "formal" settings and hasn't made a habit of it. It's a mark of respect as far as they see it, but he hasn't exactly been trained to respect. It's a very distinct directors choice in order to show the character as young, juvenile, and perhaps a little rebellious.
I've seen different expressions of the rule (or omissions), but I've seen a lot of references to men being allowed to keep wearing a hat in a diner or at a lunch counter.
(I can speculate that it was because there was no where to place the hat: no hat-check girl, might not have had hat-racks, might not have had clips on the seat or the area, might ordinarily be too full to have a place to set it down. But that's speculation without data.)