Adapted from an earlier comment of mine:
The pardon was not popular. Donald Rumsfeld (who was in Nixon’s cabinet) mentions “angry calls, heavy and constant” to the White House switchboard. Ford went from a 71% to a 50% approval; in June 1976, only 35% of people said the pardon was the right thing to do compared to 55% who said it was the wrong thing to do.
The 1976 Ford-Carter race was fairly close (297-240); while historians must always be careful with such statements, it is not unfair to say the extra handicap of the pardon lost him the election.
Eventually opinion shifted; by 1982 it was evenly split and polls after tilted towards approval.
Note it's quite possible a prosecution would never have taken place even without a pardon. Richard Ben-Veniste (an employee of the special prosecutor) later wrote:
Some, including Special Prosecutor Leon Jaworski, were of the view that Nixon's precipitous fall from the highest office was punishment enough.
...
Rumsfeld’s quote is from his recent book on Ford, When the Center Held.
Probably a dumb question, but why didn’t Congress do anything?