It depends on which physicists you are talking about, and where. Different types of physicists, and different nationalities of physicists, changed the reception.

In general, though, one can say that the one part of Einstein's 1905 work that was promoted with great interest initially was Einstein's explanation of the photoelectric effect, because this opened up new avenues in quantum theory, showing that Planck's quanta was not just a heuristic. This is what got Max Planck interested in Einstein's work, which is why it got published in Planck's journal in the first place and taken seriously — without Planck's support, Einstein (then an outsider) would have had a much harder time being taken seriously.

The other aspects of SR, such as space and time dilation, were ignored by most physicists (they are clever geometrical manipulations but it wasn't obvious they are real effects), though there were a few whose work overlapped sufficiently with Einstein and who took his work up with other work on similar topics. Those who would have denied Einstein though mostly just ignored him; SR didn't seem to have huge implications at that point for the rest of physics. When it was referred to, it was considered the Lorentz-Einstein theory of electrodynamics, a clever re-derivation of the Lorentz equation.

Interest started to increase though around 1908. Other physicists and mathematicians began exploring the implications of SR, notably Hermann Minkowski, a former professor of Einstein's (which is mostly coincidental — they didn't care for one another). Minkowski basically reformulated Einstein's work in geometrical terms (worldlines, spacetime), which initially Einstein disliked but he soon came to embrace as a core way of formulating his work. There were also some Cambridge physicists who happened upon Einstein's work and saw it as complementary to their own re-thinking of mathematical understanding of physics.

All of which is to say that the reception of SR was pretty flat initially. During this period he began working on GR, after being asked to write a review article on his own work in 1907. By 1915 he had, in collaboration with his mathematician friend Marcel Grossman, developed the fundamental equations of GR. The problem with reception, at this point, was the fact that World War I was ongoing and scientific communications were pretty stifled (and heavily politicized: British physicists were boycotting German physicists and vice versa). But even during the war there were a few physicists who learned about Einstein's work, mostly because they overlapped with Einstein in some way. These included Vsevolod Frederiks, Karl Schwarzchild, Erwin Freundlich, and Willem de Sitter, the latter of whom Einstein himself tutored in GR.

De Sitter was in contact with Arthur Eddington, in Britain, and over the course of several months basically made Eddington into a major convert to GR. Eddington hatched up a plan by which he would, when the war ended, use GR as an excuse to "heal the wounds of war" — he would test something that Freundlich had hypothesized, that GR could be confirmed by taking careful photographs of stars near the Sun during an eclipse. Eddington, a British Quaker and conscientious objector, got funding to go to the island of Principe in 1919, in order to prove the theory of a German Jew. He also widely publicized the results, resulting in a frenzy of media activity about how Einstein had overturned Newton. This is when Einstein becomes a well-known world figure, and when relativity became something that physicists couldn't keep ignoring.

The result was indeed a huge backlash from some. There were many detractors. The 1919 observations were "messier" than they are often portrayed, and in any case most scientific observations leave a lot of room for criticism and doubt. Even before 1919, in Germany they were anti-Einsteinian physicists who associated relativity with cultural evils, and — not coincidentally — with "Jewish thinking." The anti-Einsteinians were led by two German Nobel Prize winners, Philip Lenard and Johannes Stark, who portrayed Einstein and the new quantum theory as "Jewish physics" in contrast to their desired "German (Aryan) physics." If you search on here for the "Deutsche Physik" movement you'll find more writings about that. They briefly had some influence once the Nazis took power in 1933 but it was not as powerful as it is sometimes portrayed.

Separate from the blatantly anti-Semitic objections to Einstein's work, there were for a long time many physicists who either again just ignored it (it just doesn't interfere with a lot of day to day physics), or disliked it for aesthetic reasons. (Nikola Tesla is one of the latter, a fact that the Tesla-worshippers tend to prefer to ignore!) Interestingly, physicists largely, over the course of the 1930s, become Einsteinians — it was a gradual process, not an immediate one, and probably a generational one (younger physicists displacing older ones). But interest in GR waned everywhere after the 1930s. It was interesting, but difficult to use in any meaningful way, and there were other exciting problems in physics (like quantum mechanics and nuclear physics). If you track the citations of people working on GR, it basically goes to near nothing in the mid-1930s, and doesn't come back into interest until the 1960s (which is its own, post-Einstein story).

The above is summarizing a lot of scholarship (and some comes from lecture notes I took while a TA at MIT in a course on the history of physics in grad school), but gives a somewhat qualitative survey of the events and general themes. "The reception of Einstein" used to be a big topic in the history of physics and there is some excellent work from a few decades ago. One book in particular that is of value is Gerald Holton and Yehudah Elkana, ed., Albert Einstein: Historical and Cultural Perspectives (Princeton: Princeton University Press, 1982). Holton (who every time I cite I feel the need to check if he is still alive — and he is, at age 97!) in particular wrote many books and articles about the genesis and reception of Einstein's work.