If you have some time and are genuinely interested, your questions can be answered in The History of American Higher Education. I like Thelin's books because he goes into the history of black/African American students, HBCU's, women in college, rural students, and other racial and cultural minorities who are all often not or underrepresented in history. He also includes a lot of primary sources and personal stories that provide a better picture of the higher ed landscape. Pretty much everyone with a Student Affairs/Higher Ed Master's degree has to read this in grad school.

In short, Harvard has always claimed to be the oldest university in America (the argument often goes back and forth with Penn). Higher Education was (still is?) reserved for the wealthy and elite so going to college in the colonial period through the end of the 20th century was a really, really big deal. You could go to college if your family had the money not only for uniforms, fees, tuition, room, and board for an entire year, but also for transportation to Massachusetts and sometimes professor's fees and room and board as well.

The expense and general misunderstanding (new education concept) of what higher education was made it extremely rare for people to go to college, which lead to Harvard's initial branding of prestige. It must be good if only the rich are doing it, right? Harvard's alumni base and endowment has grown through the centuries, only perpetuating their prestigious image.

Havard's extensive history ties in with the other comments on their ability to attract modern day prestige and wealth, and maintain the brand.