Abe Fortas was asked to resigned by Earl Warren instead of facing impeachment. He was even put forth as a potential Chief Justice nominee, but later became embroiled in several ethics controversies regarding possible bribes. He resigned in 1969.
William Douglas also came very close to impeachment in the 50s and 60s. Twice Congress initiated impeachment proceedings, but twice Congress failed to formally impeach. Douglas resigned from the court about five years after the second impeachment attempt.

The only successful impeachment (i.e. the formal process of charging the official with misconduct) was of Associate Justice Samuel Chase, who was acquitted in 1805. This case is actually quite important, because Chase had not committed a crime. Impeachment was pursued because his political opponents had not liked some of his decisions. The fact that the Senate acquitted him, with some of his opponents voting in favor of acquittal, is a watershed in American history and the history of Judicial Review. Since then, most judges who are impeached are impeached because of criminal or ethical violations, not for political reasons (i.e. I don't like how you voted, so off with you!)

To my knowledge, all other justices of the USSC had either retired voluntarily or died in office.

edit = grammar/syntax

What's fascinating is that justices who were personally reprehensible--and maybe even monstrous--were rarely the object of consideration for impeachment.

Justice James Clark McReynolds, for example, sat on the court for three decades at the start of the 20th century--and was a bigot who refused to sit beside Justice Louis Brandeis for Brandeis’s first three years on the Court because Brandeis was Jewish. (They couldn't even take the usual group portraits of the justices, since the order of seniority would require that McReynolds and Brandeis sit beside one another.) When a woman argued a case before the Supreme Court, McReynolds left the courtroom; when a black attorney entered the court, McReynolds turned his back to signal his disapproval. He was an avid duck hunter, and he used to use his law clerks like hunting dogs, sending them running into freezing bogs and marshes to collect the fallen birds. (When asked why, he explained that the clerks were cheaper than canines.) His time on the court is littered with accounts of him going on obscene rants against attorneys, clerks, and his colleagues: everyone who worked with him agreed that he was a self-absorbed, borderline sociopathic bigot.

It's interesting that someone who was appalling in so many respects was never seriously considered to be a threat to the American public in this kind of position of power.

Source: Ian Millhiser's Injustices: The Supreme Court's History of Comforting the Comfortable and Afflicting the Afflicted.