The answer, like many details in the history of physics, is sort of a "yes and no." The issue is whether Einstein really considered what we would today call "black holes." He did, in October 1939, publish an article in the Annals of Mathematics arguing that certain types of "Schwartzchild singularities" were unlikely to exist — but what he was talking about is not exactly what we today call black holes.

Karl Schwartzchild had worked out in 1916 the exact solutions to certain of the equations for General Relativity, and had found that under certain conditions you got very odd results when you looked at stars. There were distances in which time went to zero, and space went to infinity. This is not how the physical world typically works — infinities popping up in physics are typically considered a fairly egregious sign of an error. He also played with some equations to show how much pressure it would take to make a star of a given mass into the size you'd need for this weirdness (the "Schwartzchild radius" as it is now known) to kick in. He did not argue they were real; he simply noted this was an interesting aspect of the equations.

Einstein's paper was essentially arguing against Schwarzschild's, not to make the argument that singularities of this sort didn't physically exist (nobody thought they did in 1939), but to argue that Schwarzschild's math was wrong for various reasons. He appears to have seen singularities not as a bizarre and interesting prediction fo General Relativity, but a possible flaw in GR — a sign that the equations were wrong, since they predicted physical nonsense. And so in his paper he basically shows that the assumptions that Schwarzschild had made don't actually work out in practice, and the singularity issue sort of disappears if you apply somewhat different assumptions and physical understandings to it.

Of course, Einstein was wrong on this (and several other issues in his life, but he was, despite all of the hype, only human). But the path to a modern understanding of black holes was starting at basically the same time he was working on that paper: J. Robert Oppenheimer was exploring with his students a somewhat related problem, and asked what would happen if a star collapsed to the size of this Schwarzschild radius. This involved quite a lot of other physical assumptions, but gives a singularity result as well. And this, it would many decades later turn out, ends up getting you to the reality of a modern-day understanding of a black hole, after Einstein's death (the term was only coined in the 1960s).

Which is to say, the thing that Einstein was arguing against in 1939 was not really a black hole — it was strange results in the math that did lead to the idea of black holes, but they weren't there yet. And he did not think his theory was wrong — he thought the idea of singularities was wrong, and that his theory, in reality, did not allow for them.

Now whether he continued to think that all the way to his death in 1955, I am not sure we have any way of knowing. Because he never wrote on the subject again after 1939. I would not say this is something that consumed him (the way that looking for a Unified Field Theory consumed him, or his disagreements about the nature of quantum mechanics consumed him); it seems to have been a case of him being somewhat disturbed by apparently bizarre uses of General Relativity, and he tried to correct them.

But the actual theory of black holes came somewhat later than all of this. So saying that Einstein was arguing against black holes is not quite right; he was arguing against an early version of the Schwarzschild singularity, which would only through several other developments turn into the idea of a physically-real black hole.

A very nice article on this by a physicist/historian of physics is Jeremy Bernstein, "The Reluctant Father of Black Holes," Scientific American 274, no. 6 (June 1996), 80-85.