I'm asking because I heard from someone that Jews weren't considered a race or ethnicity before Hitler's campaigning against them. I highly doubt this and want to know the truth.
You're right to doubt that. I've discussed before the interesting intersection of religion and early concepts of race in the early Spanish empire. Jews didn't stop being Jewish just because they converted- they were held under much more suspicion than an equivalent Protestant or Native American convert and were the focus of inquisition to test their new faith.
It's sort of both. You can be Jewish and not religious if you come from an ethnically Jewish background (of which there are several) or you could come from a non-Jewish background and religiously convert, which makes you automatically Jewish.
I think the answer to this will change wildly depending on who is doing the considering. Also, I usually find the term "ethnic" really vague. If you adopt a culture do you share the "ethnicity"? When does an immigrant change their ethnicity? Am I Irish because my grandparents were or am I American because they emigrated here? When did that change from Irish to American occur?
Judaism is a religion. Does practicing that religion make you a Jew? What if you are a convert (immigrant, for the purposes of comparison)? Are you different from someone that was born into the religious group?
I can answer some of these questions for the ancient world.
Ancient religion was rooted to specific places. For example ancient writers often say "the people at [certain place] have these religious practices." For example, the people of Athens have an annual celebration where the parade a giant dress through the city and adorn the statue of Athena with it. That is a religious practice linked to a specific place. No other cult to Athena (and most cities had a cult to Athena) did that form of worship.
This means that ancient people identified religion not as a category in itself but as part of a national or civic or imperial identity and set of practices. However, these gods and practices could be easily transported by migrants, so you see cults to Serapis (a Hellenistic Egyptian hybrid god) spring up in Italy and all sorts of other mixes. Judaism was one cult among many, but most ancient people would associate that cult with a nation of people from the Levant.
Judaism was, however, required a lot of work to be a member. You had to change your lifestyle, quit partaking in sacrificial meals, and get circumcised (for the men). The result was that Gentiles would go half way. They would mill about outside of synagogues to here the preaching without becoming full members of the community.
These requirements were discussed a lot by Peter and Paul who debated whether they were required for followers of Jesus's teachings. These men considered themselves and their converts Jews practicing in a particular sect of Judaism that believed that Messianic prophesies had been fulfilled. In fact, there were many early Christians (specifically Ebionites) who practiced Christianity but also kept Jewish holidays like Yom Kippur.
I'm afraid I haven't really answered your question, but I have complicated it. "Jewish" identity had two meanings: "a person from Judea" (more ethnic) and "a person who practiced Judaism" (religious). When Tacitus claims that "a sect of Jews worshiping Christ" were punished for the burning of Rome, is he talking about a group of people originally from Palestine or Roman members of a religious sect? Obviously the latter, so it seems to be used more as a religious identifier by the Romans.
Helen Fein argues that modern antisemitism is the result of anti-Judaic tropes and beliefs being combined with the "scientific racism" of the 19th century.
During the 19th (really starting in the late 18th) century, Jews were emancipated and joined the middle class and bourgeoisie; Gentiles saw this as a threat to their own social and economic mobility and dominance. The anti-Judaic stereotypes of Jews as antisocial were combined with the "science" of racism; Jews were said to be inherently biologically different (and inferior) from, say, Germans, based on studies by biologists and medical doctors.
Source: Helen Fein: "Explanations of the Origin and Evolution of Antisemitism" in The Persisting Question: Sociological Perspectives and Social Contexts of Modern Antisemitism, 1987