The question as to what causes racism goes beyond the scope of historical study alone and is still something of a contested issue. There are pertinent questions in psychology, sociology, political science, philosophy, and anthropology regarding the origins of biases based on ethic divisions, physiological traits, nationality, or other factors. After all, race is a fluid social construct that at different times is defined by different mixtures of such factors so as to classify people into specific racial groups and subgroups. “Races” as we may conceive of them today, are merely the current iterations of this sort of thinking. We can even see how current societal perceptions have continued to change conceptions of race in recent times. For example, the people of the Middle East, including those of Arab ethnicity, have traditionally been classified as part of the Caucasian (white) race, but in the wake of 9/11, white people in Europe and the U.S. increasingly perceived of Middle Easterners as constituting an “other.” This led to Arab peoples being perceived as a different race out of bigotry, but also out of a sense of protectionism against bigotry. Similarly, many people from Central and South America who were considered “white” in their country of origin have found that in the United States they are generally seen as non-white. This is due to several decades of immigration to the United States from those countries being a political issue and the immigrants generally being of a different economic class, religious denomination, skin tone, and speaking a different language than the majority of white people in the United States.

One of the best works I can recommend on the origins of racial hierarchies within the European colonial powers is The Invention of the White Race by Theodore Allen, which comes in two volumes. Allen takes the view that race originated as a tool for the powerful to divide the lower class and implement a form of social control in the colonies. It is in ways an expansion on the way that president Lyndon B. Johnson, something of a reformed racist himself, once described racism’s roots: “I’ll tell you what’s at the bottom of it. If you can convince the lowest white man that he’s better than the best colored man, he won’t notice when you’re picking his pocket.” In his books, Allen compares the formation of racial hierarchies in the New World colonies to the religious hierarchy established in Ireland by British colonialism, distinguishes it from stereotypes and ethnocentrism in pre-colonial England, while also comparing the differences in how racial hierarchies operated as between the different New World colonies. As I said, I consider it one of the best texts out there on the subject for its detail and historical analysis. That said, while Allen posits great arguments for his central thesis and showcases how racial classifications have been used, created, modified, and enforced by the powers that be, I’d say is still falls just short of being a comprehensive answer to your question of whether race was more of a post hoc (or even ad hoc) justification. When it gets down to it, there seems to always be a question as to whether the concept of race was consciously invented for these purposes, or whether the concept of race is rooted in personality traits certain people hold such as a lack of empathy and a propensity to judge and categorize the people and world around them, with the societal manifestation of racial thinking being a mere amplification of that.

As for your question about the Irish, Italians, and others accepting white identity, there’s a story for each group, though it should be noted that they weren’t really seen as non-white, just a lesser form of white. 19th century racism had a much stronger emphasis on “sub-races” than what we see today. Nevertheless, I would recommend Noel Ignatiev’s book How the Irish Became White. But the basic idea is that immigrant groups assimilated into the white identity so that they could benefit from white privilege. They wanted to be part of the higher rungs of society or at least to be spared from some of the hardships that non-whites or lesser whites faced in the U.S. It is a truth throughout history that when something serves as a nexus to power or higher social status, people want a part of it. Their lives and livelihoods may even depend on it. That’s why the Irish wanted to solidify their white identity. That’s why many second+ generation Hispanic Americans are increasingly identifying themselves as white even when they’re not seen as such. That’s why peoples conquered by the early Muslim Caliphates converted to Islam, or why Roman citizens became Christian once the Emperors favored the religion. It’s even why people buy the clothing brands that celebrities wear, or why Patrick Bateman cared so much about business cards in American Psycho. There are social, sociological, political, and plainly practical reasons for doing so, especially when in an environment where it has a real material impact on one’s life.

Edited for typos.