They are the oldest, and so have the longest histories in which to grow their prestige. They were also exclusive. For the first century or more of operation, the schools only admitted students who fit the criteria. Namely, that they were male, white, and wealthy. Women, people of colour, and poor white men, were all excluded from the Universities.

Harvard and W&M were both set up in the 17th century(1636 and 1693 are their respective founding dates), while Yale and the University of Pennsylvania were founded in the 18th century(1701 and 1755). In this time period, education was only given to a minimum level, and after that, further education was a pursuit of the wealthy. I'll look specifically at Harvard, as it was the first University, and Massachusetts was the first colony to enact compulsory educational legislation.

Massachusetts was the first colony to pass a piece of legislation governing this, and theirs stated that the minimum requirement was to be able to read. In their words, it was thus:

and for this end they[selectmen], or the greater part of them, shall have power to take account from time to time of their parents and masters, and of their children, concerning their calling and impliment of their children, especially of their ability to read and understand the principles of religion and the capital laws of the country

The selectmen referenced were "the chosen men of every town", which I am guessing refers to respected figures in the community. They were given the right to check the literacy of the children, and if they were not up to standard, further measures including fines, and removal of the children into apprenticeships, could be taken.

It is interesting to note from the quoted passage of law that the reason was so that they could "understand the principles of religion and the capital laws of the country", or in other words, be good Christians and citizens.

The apprenticeships these children or young people were moved into were for skilled labour. The purpose of the educational act above was to provide skilled labourers for the colony, as it was assumed that unskilled citizens would become paupers and be a greater burden on the state(which was most likely true).

So we have here, for the most part, the citizens of the Colony of Massachusetts(where Harvard was situated), being educated to the level of being able to read, but no further, and then entering skilled apprenticeships.

Harvard was the alternative here. But who could go there? Certainly not women, poor whites, or people of colour. So it was left to the sons of the colony elite to attend. These were men that came from families educated previously at institutions in the UK, like Cambridge and Oxford(also on your list). They were wealthy families, who had been able to afford higher education for their children for generations.

Harvard's enrollment process was discriminating right through until the 20th century. After World War I when millions of immigrants arrived in the United States, Harvard imposed quotas on Jews, allowing only the most talented students in. Other groups less desirable were also less likely to gain access. Those most likely to get in?

Students of good character who could pay the tuition were usually admitted, even if sometimes on academic condition. Sons of men of inherited wealth, the old college families, were joined by sons of nouveaux riches businessmen.

These sons of the elite, who were descended off "the old college families", were to continue in the highest levels of society. They were America's upper class, and as such formed the bulk of the political and economic elites. Harvard, along with Yale and Princeton, had educated "proportionately more leaders than any other undergraduate colleges in the United States", lending a great deal to their prestige.

Bonus material from Wikipedia - a list of Presidents and where they went to university.

Out of 43 Presidents, there were: 7 who attended Harvard in some form, 5 for Yale, 4 for W&M, and 3 for Princeton.

So those four schools alone educated just under half of the men who went onto become President. If you go to the individual college alumni pages, you'll find a bevy of other political figures who went through those institutions too.

Sources:

Marcia G. Synnott, 'The Admission and Assimilation of Minority Students at Harvard, Yale, and Princeton, 1900-1970', History of Education Quarterly, found at Jstor.

Yolanda T. Moses, 'Race, Higher Education, and American Society', Journal of Anthropological Research, found at Jstor.

Marcus W. Jernegan, 'Compulsory Education in the American Colonies: I. New England', The School Review, found at Jstor.