Part of the problem in answering this question is that it is culturally loaded. For example, if you think the expansion of European cultural forms, technology, and people is good then you could say it was an upside. If you think the destruction, modification, or coercion of native culture is bad then colonialism is detrimental.

As someone who studies and teaches about the early colonial period in the Americas, I think that the comparison to Fascism and Hitler is unwarranted or at least a bad analogy. For example, the massive death of Native Americans was almost entirely due to the arrival of Eurasian disease, and since there was no germ theory or real knowledge of contagion the Europeans had no idea how it was spreading nor that they were the cause. In the case of Hitler, he consciously initiated a plan to exterminate certain people. In reality, the Spanish did not want to kill Native Americans they wanted to keep them as laborers.

Moreover, the forces which shaped early expansion into the Americas were not dependent on the political system, which was in no way like Fascism. Most early conquest and exploration in the Caribbean was privately funded and carried out. The Spanish monarchy did not dictate where, when or how conquests occurred. For a good overview of the Spanish conquest see Restall's Seven Myths of the Spanish Conquest