I'm going to go on a limb and actually attempt to answer this question despite people's objections over the definition of prestigious.

Harvard's prestige may have a large part to do with how old it is. For much of colonial New England's history, if someone of means wanted to get a higher education, the only options were to go to Harvard or back to England.

For obvious reasons, Harvard become a popular option for the rich and powerful of New England society. This cycle feeds itself as new up and comers in the system attempt to form connections with the existing establishment.

Both Roosevelt presidents can claim Harvard as their alma mater. The Roosevelt family is the closest you can come to naming blue blooded American royalty. The shear number of American and world leaders, justices, business men, etc that have passed through Harvard over the centuries is an inertia that is difficult to overcome.

This also feeds into another of Harvard's inherent advantages in the prestige department: their endowment. Because of the aforementioned individuals, Harvard gets more donations and can quite literally afford to acquire anything they want. The best facilities, or outbid on acquiring world class faculty. Harvard has a larger endowment than any other American institution of higher learning. On paper, their endowment also dwarfs Oxford's, however Oxford's other assets are more difficult to access.