California is interesting demographically, along with other states like Florida and Nevada, for their consistent high population growth ever since statehood, leading to large potential student populations.
One must be careful of where they obtain these period rankings of universities, making sure they were actually published at the time, instead of a modern ranking of how the author perceived the institutions to have performed in the past, potentially colored by other modern assessments of their reputation and performance. Moving away from the 1920s, the concept of widely-disseminated, "universal," lists of college and university rankings is also relatively recent. In their 2009 article Rating the Rankings, Wendy Espeland and Michael Sauder explain that prior to the 1980s, compact lists of comprehensive comparative rankings of U.S. colleges universities were not widely available to the general public, being produced only for the information of "academic insiders." This changed in 1983, when U.S. News and World Report published their first set of comprehensive college and university rankings. Other publications soon followed suit, and this system has even grown to have a small subfield in academia dedicated to studying it and its effects on the psychology of institutions.
In their 2010 article U.S. News & World Report College Rankings: Modeling Institutional Effects on Organizational Reputation, Nicholas A. Bowman and Michael N. Bastedo determined that "published college rankings have a significant impact on future peer [i.e., fellow institution] assessments, independent of changes in organizational quality and performance and even of prior peer assessments of reputation." Such is the case of Oberlin College:
In 1983, when U.S. News & World Report published its first set of college rankings, Oberlin College ranked fifth among the nation’s liberal arts colleges. This was hardly surprising as the ranking was based entirely upon reputation among its peers, and Oberlin had a long and storied history. It had a particularly strong reputation for placing students in top graduate programs, for decades ranking first among liberal arts colleges in the number of students who ultimately earned the doctorate (National Science Foundation 2006). Four years later, in 1987, Oberlin remained in the fifth position; the next year Oberlin was stunned to fall completely out of the top 10. Beginning in 1988, the U.S. News rankings methodology began to include quantitative measures of student and institutional characteristics that it believed were more objective measures of quality, and consequently peer reputation became a progressively smaller part of the overall ranking. Oberlin did not score at the top of these new measures, and its ranking suffered. By the mid-1990s, Oberlin was in serious danger of falling out of the top 25, which U.S. News defined as the “top tier” of liberal arts colleges. More curiously, Oberlin’s peer reputation suffered during the period as well, over time moving closer and closer to its overall ranking, despite having changed little over the years. Was Oberlin no longer as good as it thought it was? Or were the rankings damaging its reputation among college leaders, even while its quality remained relatively unchanged?
In their 2011 article Anchoring Effects in World University Rankings: Exploring Biases in Reputation Scores, Bowman and Bastedo hypothesized that the anchoring effect has a major role in the determination of college and university rankings, and then came to the conclusion that it actually does have a strong effect on future university rankings after the release of the initial list. The anchoring effect is a psychological concept wherein a subject depends heavily on an initial piece of information (in this case, an initial set of college and university rankings) in the course of making future decisions (future releases of institutional rankings), allowing well-established, frequently very old, institutions to cement their places on lists. This works against newer, less-established institutions, who must first develop a reputation among their peers, as well as among the general public and potential students, that includes a strong track record of research and sending a high percentage of students to, and preparing them to successfully complete, graduate and post-graduate programs.
"Universal" rankings systems that try to place a variety of institutions on one list can also be harmful because of pigeonholing. Citing a previous article by Kimberly D. Elsbach and Roderick M. Kramer (Members' Responses to Organizational Identity Threats: Encountering and Countering the Business Week Rankings), Bowman and Bastedo also contended that the prestige of achieving a certain place in a list can lead to a "crisis of institutional identity." Espeland and Sauder said that this can cause institutions to completely change their academic and research focus in order to compete with others in yearly rankings; journalist Peter Sacks explained the "dangers of using standardized or even universal metrics to evaluate schools doing fundamentally different jobs of offering specialized forms of education" and that "judging schools according to a single set of criteria ignores the fact that schools have different aspirations and punishes those with distinctive or non-elite missions." These rankings can also have an effect on student behavior in selection of institutions to attend, as they act almost like consumers of a product.
As many in higher education believe that the U.S. News rankings are shallow or even misleading (McDonough et al. 1998; Thacker 2005), and acknowledging the widespread evidence that rankings have significant effects on college choice at both undergraduate and graduate levels (Bowman and Bastedo 2009; Martins 2005; Monks and Ehrenberg 1999; Sauder and Lancaster 2006), the additional institutional effects of U.S. News on expert opinion are particularly significant. From a methodological perspective, this effect may distort the quality of the reputation scores that are used in calculating the rankings. The more that the reputations of colleges are affected by the ranking, despite other evidence to the contrary, the more the U.S. News rankings become college reputation. Reputation scores are thus likely to have decreased value over the long run.
Sources:
Bowman, Nicholas A., and Michael N. Bastedo. "U.S. News & World Report College Rankings: Modeling Institutional Effects on Organizational Reputation." American Journal of Education 116, No. 2 (February 2010): 163-183.
Bowman, Nicholas A., and Michael N. Bastedo. "Anchoring Effects in World University Rankings: Exploring Biases in Reputation Scores." Higher Education 61, No. 4 (April 2011): 431-444.
Elsbach, Kimberly D., and Roderick M. Kramer. "Members' Responses to Organizational Identity Threats: Encountering and Countering the Business Week Rankings.” Administrative Science Quarterly 41, No. 3 (Sep., 1996): 442-476.
Espeland, Wendy, and Michael Sauder. “Rating the Rankings.” Contexts 8, No. 2 Higher Ed in Uncertain Times (Spring 2009): 16-21.
Without knowing which particular lists you're looking at, I suspect the universities you're thinking of that keep reappearing are Harvard, Yale, Princeton, etc. In effect, what you're seeing is the replication of privilege in American higher education. (I answered a question about college in 1920 here that provides some additional context.)
To take it way back to the beginning, higher education in the American colonies is generally recognized as having started with a small endowment and donation from John Harvard to the Massachusetts Bay college that was later renamed in his honor. The school served young men (all White, save a few Indigenous men connected to the Harvard Indian College, until the 1900s) and operated more like a boarding school than a college as we think of it today. The average age of students was around 15 1/2.
As the various colonies got their feet under them and developed a need to formalize education systems, men in positions of power or access to power (i.e. non-disabled White men) in early America created spaces for formal education for their sons. So, just as Boston gave rise to Harvard, New Haven men established Yale, Princeton men established Princeton, etc. In total, nine institutions were established during the colonial era. They are:
A primary goal of these schools was networking and they served their purpose. Classmates went on to found law firms, businesses, and work in government together. Subsequent generation of men would send their sons to the same institutions they went to so that their sons could network with the sons of other men in power. This pattern continued well into the 1900s; men who made the lists were men in power, or had access to power, and they associated "good" with where they went and where they were planning on sending their sons. It became a reinforcing feedback loop: men in power went to these particular schools so if a young man wanted a position of authority or power, he would attend one of those schools.
To be clear, however, these Colonial Colleges were not the only institutes of higher education. Colleges for young women were established but they served a different purpose. A young White man went to a Colonial College because he was destined (or his father thought he was) to be a great man. A young White woman went to a woman's college because her future son was destined for greatness. Free Black men and women, simply did not attend the colonial colleges. They instead, attended schools like Oberlin, which quickly established itself a rigorous college for non-White male (and female) students as it followed a classical curriculum not unlike that seen at Harvard. While Oberlin did enroll white male students, they were more likely to lack political connections and specific education required to get into the colonial colleges. A man with more humble aspirations, or more limited means, might go a one of the land grant colleges that were founded in various waves during the 1800s. To be sure, not all of the Colonial Colleges are held in the same high esteem in the modern era which has a lot to do with endowments and how various states support colleges within its borders.
This relationship between "good" and "exclusive" and how power traditional works in this country can also shed some light on why so few Historically Black Colleges and Universities (HBCUs) or Tribal colleges are seen on such rankings even when their graduates would go on to found entire new fields of history, science, or mathematics. As a general rule, if something related to American education is deemed good or better than, we're likely to see the presence of men with access to power.
Keep in mind the colonial colleges were founded by men in power to prepare sons and their friend's sons access to the networks and knowledge they had, which means as women began to attend college in greater and greater numbers, it was about them gaining access to male spaces. As more and more students of color began going to college in the 1970s, especially those who attended Primarily White Institutes (PWIs) , they were stepping into White spaces. Nearly all of the Colonial Colleges had their own quirky admissions exams until well into the 1900s. This meant that if a student wanted to attend Yale, he had to memorize a very particular set of Greek and Latin passages. And they could easily refuse to admit a young man of color or steer a young woman over to one of the Seven Sisters, the legacy women's colleges.
While the creation of the SAT exam was an attempt to even out admission policies among the "best" colleges, they all continued to give extra weight to legacy students. In effect, this was codifying the feedback loop I mentioned before. So, some schools are locked into performance on lists due to a combination of factors related to perception, exclusivity, and tradition. When "ranking" schools, publishers would look at how hard it was to get into, what graduates did after graduating, and other variables. In effect, it's not so much that newer colleges are kept from getting on to the list, it's more that the weight of tradition, exclusively, and access to power among those who've always been on the list has been too hard to overcome.