The best answer is that nobody knows for sure, or at least the various biographers do not agree on the reason.

The most common answer, but not verifiable, is that certain Mafia figures (possibly Lansky or Costello?) had proof that Hoover was a closet homosexual, or even a secret cross-dresser. The known fact is that Hoover cohabited for decades with FBI agent Clyde Tolson. Both men were single; neither was known to have a long-term relationship with a woman (although Hoover had gone on 'publicity' dates with Hollywood starlets). Hoover bequeathed his estate to Tolson upon his death. Also there have been numerous reports that one or more photos exist showing Hoover dressed in drag.

It may seem far-fetched to think that a Mafioso could blackmail Hoover; however it should be kept in mind that Hoover himself is generally thought to have blackmailed several Presidents, including Kennedy and Nixon. This would have been the reason why Hoover was the only highly-placed member of the Executive branch of the US Government to have had a 'lifetime' tenure in his post. So blackmail as a device of control certainly was effective.

Hoover's interests shifted in the 40s and 50s from busting bank robbers to more political matters of domestic stability. In particular, he was a firebreathing anti-Communist and social conservative and focused on that to the exclusion of mob matters. However, I'd question the premise of your question: Hoover refocused on crime after the 1957 Apalachin meeting made a mockery of the FBI. The concentration would never be as consistently strong as it was in the 1920s and 1930s, and it was on a relative low in 1963 when Kennedy confronted him, but Hoover never completely ignored the issue again after 1957.

I don't think it's quite fair to say that Hoover was a far-right sympathizer so much as he was radically against anything that could subvert social decency and order, or at least as defined by the Director-he also happily carried out domestic intelligence operations against Nazis at FDR's behest and would break the KKK's back under Johnson, in spite of his rampant distaste for the Civil Rights movement. But it's quite plausible that his anti-Communism made him see the Mafia as a lesser evil, or at least a lesser priority, than organized crime throughout the 1950s. The Mob certainly didn't like the Commies-they'd be out of business under them.

(And to be fair, Hoover had better relationships with Roosevelt and Johnson than he did with any other Presidents, so that could have played a role. If Roosevelt wanted to indict the head of Goldman Sachs, Hoover would have done it immediately. Hell, what FDR would say about HRC doing speeches in front of them... he'd have Hoover on it immediately.)