Catholics, including American-born Catholics, used to be discriminated against in the US, but how did people know who was Catholic and who wasn't? Was it only based on who they associated with? Were there any noticeable physical differences between Catholics and Protestants?

Their discrimination would have been based on two clues.

Firstly, where you came from and who you associated with. There were various parts of cities in the US, such as Hells Kitchen in Manhatten, or East Boston that were notorious slums filled with Irish Catholics. Often poorer parts of cities were filled with Irish immigrants which in turn fueled the stereotype that they were bad people.

Secondly, your name. If your name was George Pennyweather and you came from Harlem, odds were you were a fairly wealthy white Anglo-Saxon protestant. If your name was Patrick James Kennedy, and you came from East Boston (As JFK's Grandfather was/did), odds are you're Irish Catholic.