It's best to think of higher education during the colonial era more like secondary school than college as we think of it today. One could almost think of them as akin to (but not exactly like) boarding schools for white boys from families of means. In 1810, the average age at a colony college was 15 and all of those students were boys. They were virtually all white (notable exceptions include the indigenous boys and men who attended Harvard's Indian College) and in most cases, they were expected to pursue a life in politics, behind the pulpit, or practicing law. The first reason not everyone went was because not everyone was allowed to go.
The word "peculiar" was often during this era to describe young women's education. White girls and women from families of means were expected to receive an education - but the frame was how their education could benefit their husband or their son. Young women didn't enroll in institutes of higher education because it was neither allowed nor necessary for their anticipated future. They were expected to become wives and mothers - not lawyers or politicians. Oberlin College, founded in 1833, would be the first American college to welcome women - white and Black. (There were some exceptions to this, most notably in Quaker communities, especially Moravian College, but they didn't award degrees until the Civil War.)
Attending college as a Black person - man or woman - simply wasn't an option. Efforts to open Black colleges in the north were routinely squashed. Even non-degree conferring schools for freed Black adolescents faced hostility. Many were vandalized, including having their wells poisoned with horse or cow manure. Meanwhile, many of the colonial colleges owned enslaved people and used their labor to establish institutional wealth.^1
So higher education was off-limits to 50% of the white population because they were female, all free Black Americans or enslaved people, and virtually all indigenous people. We can, though, look back at the white boys and young men who did go. The second reason not everyone went was because many young men didn't need what the schools offered.
The colony colleges
had what is described as a classical curriculum - Greek and/or Latin, some math, and some sciences. it was a deeply academic education - in effect, useless in any meaningful sense. The prevailing belief was that a young man would best be prepared for his future if he had a rich, esoteric education. Parents who sent their son to one of the colonial colleges (or young men who enrolled on their own) most likely saw a future as a lawyer, politician, or clergy member. In many cases, it would be because that's the career the father had, but not always. Once enrolled, the young men wouldn't study the law or politics as one would today as a law student or political science major. Instead, young men would read, talk through, and memorize information they would then recite back to their professors to prove they learned it. This isn't to say all they learned was useless once they left - they developed analytical, communication, and logic skills. At some schools, they would learn a second language like French and practical skills like architecture. Higher education was also a place for these young men to network. They would often meet up with other like-minded men and go on to form law firms and political alliances.^2
In most cases, young men from trade or merchant families would apprentice with their father or relatives. Sometimes, families saved enough to send a younger son. In others, a young man from an improvised background could find someone to pay his tuition. Young men, though, without family means to pay for classes and books rarely went simply because their path was set and it didn't require knowing Latin or Greek.
This response goes into more detail about the college application process.
Wilder, C. S. (2014). Ebony and ivy: Race, slavery, and the troubled history of America's universities. Bloomsbury Publishing USA.
Wright, C. E. (2005). Revolutionary generation: Harvard men and the consequences of independence. Univ of Massachusetts Press.