This is a very complex topic, but I'm going to attempt to provide (at least the groundwork of) an answer. This is also my first post here, so I welcome any correction or modification.

I'm going to start with your point about "Survival of the Fittest" because that was a great motivating intellectual factor in Imperialism. The evolutionary theories (most importantly for us, natural selection) of Charles Darwin had become quite widespread by the late 19th century. As you alluded to, political and social theorists adapted his theories to politics and society, manifesting itself in the idea that stronger, more powerful nations had the authority to subjugate weaker ones. As the nations that were engaging in these imperialistic ventures were European and (albeit a bit later) American, a racial component to this came to be prevalent. This idea is manifested intellectually in Rudyard Kipling's famous poem "The White Man's Burden", first published in 1899. "The White Man's Burden" put forth the idea that white men were morally and intellectually superior to other races, and that it was their responsibility to subjugate them in order to guide them towards living in a productive, peaceful, Christian society (and of course exploit their resources, labor and markets, but this is not explicitly stated by Kipling). Kipling's poem is certainly not the cause of European and American Imperialism, but it is a reflection of the prevailing attitude of political leaders at the time and the nominal justification they used for building these empires.

The end of the Imperialist era is commonly agreed upon to be World War I. World War I was a social and political realignment in a plethora of ways, but in the sense of Imperialism it takes on a few forms. As far as shifting attitudes, Wilson's Fourteen Points is the most prominent example of this. One of the key proposals of the Fourteen Points was the notion of self-determination for each nation, that is the right of a people to rule themselves. This was a stark contrast to the commonly accepted "balance of power" politics in vogue at the time. Wilson's aspirations of ensuring self-determination did not take hold in the Versailles Treaty in a practical sense (the European powers still concerned themselves with the creation of large buffer states comprised of multiple ethnic groups to act as counterbalances to other nations) but its intellectual thrust still had quite a resonance with people who considered themselves unfairly subjugated (it's worth noting here that a young Ho Chi Minh travelled to Paris in 1917 and was greatly excited by Wilson's stated goals).

Moreover, I know that in American high school, the acronym MAIN (Militarism, Alliances, Imperialism, Nationalism) is used in identifying the chief causes of World War I, but I am not sure how old this usage is. It nevertheless directly blames Imperialism for the start of World War I.

After World War I, even though a repudiation of sorts of Imperialism by Wilson was widespread, the victorious European nations and United States still retained their overseas possessions. The intellectual force behind Imperialism I identified before (Social Darwinism) was still strong. So strong it was that it was used as a cornerstone of the intellectual and political movement that rose up in the 1920's: Fascism. Fascist theory aggressively applied Social Darwinist principles to governance and politics, stating that nations were in perpetual competition with each other and that stronger nations overtake weaker ones. It dispensed with the morality of "White Man's Burden" and instead romanticized the subjugation and exploitation of weaker peoples for the benefit of stronger ones. These ideas, of course, gained hold in Italy and Germany, but there were prominent Fascist movements in other European nations.

World War II is of course the most destructive conflict in human history and was directly caused by German aggression. This reality completely discredited Fascism and, by extent, Social Darwinism. Moreover, the unchecked racist policies of the Nazi government led to the systematic mass murder of Jews and other minority groups in the Holocaust, something which shocked the world. The racism which Social Darwinism produced was also thus soundly repudiated. With the concept of "White Man's Burden" being out of fashion for some time, the destruction of Social Darwinism as a viable intellectual force extinguished the ideological foundation for Imperialism. Thus, imperialism comes to be associated with racism, exploitation, and violence which is our contemporary understanding.

Even though some nations still attempted to control others for a time after World War II, (France in Algeria and Indochina comes to mind) The United States dispensed with some major overseas holdings (the Philippines in 1946) and those that it kept were incorporated in a more democratic sense (Hawaii gained statehood in 1959). Moreover, India (the so-called "Jewel in the Crown" of the British Empire) gained independence from Britain in 1947. African nations also steadily started to gain independence from their European suzerains throughout the 1950's and 60's (largely peacefully, with notable exceptions for Algeria and the nations under Portuguese control). Wilson's notion of self-determination had taken hold. Even though nations still engage in interventionist wars, as you say in your question imperialism is anathema to the vast majority of people and the historical era it defines is associated with exploitation, racism, and violence.

Surely I am missing a few key points that perhaps a specialist can provide, but I think this is a good representation of the narrative. As I said before, this is my first post here so any modification or correction is welcome.